Jul 24 15:45:24.650378 ip-10-0-140-155 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 15:45:24.650392 ip-10-0-140-155 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 15:45:24.650401 ip-10-0-140-155 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 15:45:24.650730 ip-10-0-140-155 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 15:45:34.900203 ip-10-0-140-155 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 15:45:34.900221 ip-10-0-140-155 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fa53a959aa6647be98360b900f104ce0 -- Jul 24 15:47:10.309398 ip-10-0-140-155 systemd[1]: Starting Kubernetes Kubelet... Jul 24 15:47:10.757728 ip-10-0-140-155 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:47:10.757728 ip-10-0-140-155 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 15:47:10.757728 ip-10-0-140-155 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:47:10.757728 ip-10-0-140-155 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 15:47:10.757728 ip-10-0-140-155 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:47:10.759627 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.759545 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 15:47:10.764451 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764438 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:47:10.764451 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764452 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764456 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764460 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764464 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764467 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764470 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764472 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764475 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764477 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764480 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764483 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764485 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764488 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764490 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764495 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764498 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764501 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764504 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764507 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:47:10.764511 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764510 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764513 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764517 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764520 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764523 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764525 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764532 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764535 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764537 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764540 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764542 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764545 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764547 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764550 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764552 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764555 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764557 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764560 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764562 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764564 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:47:10.764951 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764567 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764570 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764572 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764575 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764577 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764579 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764582 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764584 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764586 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764589 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764591 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764593 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764597 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764600 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764603 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764605 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764608 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764610 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764613 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:47:10.765463 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764618 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764621 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764624 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764626 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764629 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764631 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764634 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764636 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764639 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764641 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764643 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764646 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764649 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764651 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764654 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764656 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764659 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764661 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764664 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764666 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:47:10.765920 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764669 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764671 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764674 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764676 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764679 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764681 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764684 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764687 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764689 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764692 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764695 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764697 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764700 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764702 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764705 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764707 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764710 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764712 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764715 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.764718 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:47:10.766408 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765423 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765429 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765433 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765436 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765440 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765444 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765447 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765450 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765452 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765456 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765458 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765460 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765463 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765466 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765469 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765472 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765474 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765477 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765480 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765483 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:47:10.766890 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765486 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765488 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765490 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765493 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765495 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765498 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765501 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765503 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765505 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765508 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765510 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765513 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765515 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765518 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765520 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765522 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765525 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765528 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765531 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765533 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:47:10.767421 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765536 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765538 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765541 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765543 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765545 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765548 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765551 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765553 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765556 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765559 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765563 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765567 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765570 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765573 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765575 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765578 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765581 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765584 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765586 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765589 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:47:10.767889 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765591 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765594 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765597 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765600 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765603 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765606 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765609 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765611 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765614 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765617 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765619 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765622 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765624 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765627 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765629 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765632 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765635 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765638 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765641 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:47:10.768388 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765644 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765646 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765653 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765656 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765659 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765661 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765664 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765666 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765669 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765672 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765674 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765676 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765679 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765681 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765684 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765686 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765689 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765691 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765694 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.765696 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:47:10.768839 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766037 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766041 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766043 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766046 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766049 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766051 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766054 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766056 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766059 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766062 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766065 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766067 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766070 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766072 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766076 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766079 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766081 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766084 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766086 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766089 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:47:10.769336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766091 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766093 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766096 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766098 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766101 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766104 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766106 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766109 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766111 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766113 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766116 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766118 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766121 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766124 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766126 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766129 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766132 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766134 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766137 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:47:10.769806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766139 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766141 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766144 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766147 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766164 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766167 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766170 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766174 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766177 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766179 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766181 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766184 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766186 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766189 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766191 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766194 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766197 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766199 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766202 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766204 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766206 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:47:10.770272 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766210 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766212 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766215 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766217 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766221 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766225 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766229 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766232 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766235 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766238 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766241 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766244 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766246 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766248 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766251 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766254 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766257 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766259 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766263 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:47:10.770767 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766266 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766268 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766270 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766273 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766275 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766278 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766281 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766283 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766286 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766289 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766291 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766295 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766299 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766301 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766305 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766307 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766309 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766315 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766317 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:47:10.771253 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766320 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766385 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766391 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766397 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766401 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766405 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766408 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766412 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766416 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766420 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766423 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766426 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766431 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766434 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766437 2580 flags.go:64] FLAG: --cgroup-root="" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766440 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766442 2580 flags.go:64] FLAG: --client-ca-file="" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766445 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766448 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766452 2580 flags.go:64] FLAG: --cluster-domain="" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766455 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766458 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766461 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766464 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 15:47:10.771704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766467 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766470 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766473 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766476 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766478 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766481 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766485 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766488 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766491 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766494 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766497 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766500 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766503 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766506 2580 flags.go:64] FLAG: --enable-server="true" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766509 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766513 2580 flags.go:64] FLAG: --event-burst="100" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766516 2580 flags.go:64] FLAG: --event-qps="50" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766518 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766522 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766525 2580 flags.go:64] FLAG: --eviction-hard="" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766529 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766532 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766535 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766537 2580 flags.go:64] FLAG: --eviction-soft="" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766540 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 15:47:10.772316 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766543 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766545 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766548 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766551 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766553 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766556 2580 flags.go:64] FLAG: --feature-gates="" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766560 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766563 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766565 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766568 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766571 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766574 2580 flags.go:64] FLAG: --help="false" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766577 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-155.ec2.internal" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766581 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766584 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766586 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766590 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766593 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766596 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766599 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766601 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766604 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766606 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766609 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 15:47:10.772916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766612 2580 flags.go:64] FLAG: --kube-reserved="" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766615 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766620 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766625 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766627 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766630 2580 flags.go:64] FLAG: --lock-file="" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766632 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766635 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766638 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766643 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766646 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766648 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766651 2580 flags.go:64] FLAG: --logging-format="text" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766653 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766657 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766659 2580 flags.go:64] FLAG: --manifest-url="" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766662 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766666 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766669 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766673 2580 flags.go:64] FLAG: --max-pods="110" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766675 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766679 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766682 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766685 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766687 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 15:47:10.773522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766690 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766693 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766699 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766703 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766705 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766708 2580 flags.go:64] FLAG: --pod-cidr="" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766711 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766715 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766718 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766721 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766724 2580 flags.go:64] FLAG: --port="10250" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766728 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766730 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e40185dd40776099" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766733 2580 flags.go:64] FLAG: --qos-reserved="" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766736 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766739 2580 flags.go:64] FLAG: --register-node="true" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766742 2580 flags.go:64] FLAG: --register-with-taints="" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766745 2580 flags.go:64] FLAG: --registry-burst="10" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766747 2580 flags.go:64] FLAG: --registry-qps="5" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766750 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766753 2580 flags.go:64] FLAG: --reserved-memory="" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766756 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766759 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766761 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766764 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 15:47:10.774119 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766767 2580 flags.go:64] FLAG: --runonce="false" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766769 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766772 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766776 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766779 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766782 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766784 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766787 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766790 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766793 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766795 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766798 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766801 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766804 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766806 2580 flags.go:64] FLAG: --system-cgroups="" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766809 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766814 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766817 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766821 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766825 2580 flags.go:64] FLAG: --tls-min-version="" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766827 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766830 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766832 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766835 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766838 2580 flags.go:64] FLAG: --v="2" Jul 24 15:47:10.774724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766841 2580 flags.go:64] FLAG: --version="false" Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766845 2580 flags.go:64] FLAG: --vmodule="" Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766848 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.766852 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766964 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766968 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766971 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766973 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766986 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766989 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766994 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.766997 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767000 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767002 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767005 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767007 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767010 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767012 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767015 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767018 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:47:10.775377 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767020 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767022 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767025 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767027 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767030 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767033 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767035 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767038 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767040 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767044 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767047 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767050 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767053 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767055 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767058 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767060 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767063 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767065 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767067 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767070 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:47:10.775881 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767072 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767074 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767078 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767080 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767083 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767085 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767087 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767090 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767092 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767094 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767097 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767099 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767101 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767104 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767106 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767108 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767111 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767113 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767116 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767119 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:47:10.776455 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767121 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767124 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767126 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767129 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767131 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767134 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767136 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767138 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767142 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767145 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767148 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767165 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767168 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767171 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767174 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767177 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767180 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767182 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767185 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:47:10.776944 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767187 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767190 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767192 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767196 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767200 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767203 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767205 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767208 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767210 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767212 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767215 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767217 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767220 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767223 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767225 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767228 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767230 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767232 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767234 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:47:10.777423 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767237 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767240 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767243 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767245 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.767247 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.768287 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.775214 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.775233 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.776269 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.776281 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776350 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776355 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776359 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776364 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:47:10.777883 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776368 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776371 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776373 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776376 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776380 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776383 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776386 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776389 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776391 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776394 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776397 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776399 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776402 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776404 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776407 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776410 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776412 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776414 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776417 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:47:10.778242 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776419 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776422 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776424 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776427 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776430 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776432 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776441 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776444 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776446 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776449 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776451 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776454 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776456 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776460 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776462 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776465 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776467 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776470 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776472 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:47:10.778695 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776475 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776477 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776480 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776482 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776486 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776488 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776491 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776493 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776496 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776498 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776501 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776503 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776506 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776508 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776510 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776513 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776515 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776517 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776520 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776523 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776531 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:47:10.779221 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776534 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776537 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776539 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776542 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776544 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776547 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776549 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776552 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776554 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776556 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776559 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776561 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776564 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776566 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776568 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776572 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776574 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776576 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776579 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776581 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:47:10.779738 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776584 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776588 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776591 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776593 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776595 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776598 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776600 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776603 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776605 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776608 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776611 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776613 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776621 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776624 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776627 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:47:10.780215 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776629 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.776634 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776779 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776784 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776787 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776789 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776792 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776795 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776797 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776799 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776802 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776804 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776807 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776809 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776812 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:47:10.780564 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776814 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776816 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776819 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776821 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776823 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776826 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776828 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776830 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776833 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776835 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776838 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776840 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776842 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776845 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776849 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776858 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776861 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776864 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776866 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776869 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:47:10.780925 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776871 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776874 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776882 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776884 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776887 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776889 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776891 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776894 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776896 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776898 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776901 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776904 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776907 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776910 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776913 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776916 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776919 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776921 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776924 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:47:10.781405 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776927 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776929 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776932 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776934 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776937 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776939 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776942 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776944 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776946 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776956 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776959 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776961 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776963 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776966 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776968 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776971 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776973 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776975 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776978 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776980 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:47:10.781855 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776983 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776985 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776988 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776990 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776992 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776995 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776997 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.776999 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777002 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777004 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777006 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777009 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777011 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777013 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777016 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777019 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777022 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777024 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777027 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:47:10.782336 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777029 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777032 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777034 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777042 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777044 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777047 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777049 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:10.777051 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.777056 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.777188 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.779312 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.780684 2580 server.go:1019] "Starting client certificate rotation" Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.780787 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:47:10.782786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.780834 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:47:10.783506 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.783493 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:47:10.789465 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.789443 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:47:10.800903 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.800885 2580 log.go:25] "Validated CRI v1 runtime API" Jul 24 15:47:10.803043 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.803029 2580 log.go:25] "Validated CRI v1 image API" Jul 24 15:47:10.803886 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.803872 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 15:47:10.806757 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.806741 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:47:10.807433 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.807407 2580 fs.go:135] Filesystem UUIDs: map[09a94214-9110-4813-a508-9dd9f716a76d:/dev/nvme0n1p4 0f4a27bd-ed0e-4cfc-b0f0-71b48576caa0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 24 15:47:10.807491 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.807434 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 15:47:10.812546 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.812446 2580 manager.go:216] Machine: {Timestamp:2026-07-24 15:47:10.811023551 +0000 UTC m=+0.369414263 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102627 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29a5fb763d53a748d7bb33ab32bbf2 SystemUUID:ec29a5fb-763d-53a7-48d7-bb33ab32bbf2 BootID:fa53a959-aa66-47be-9836-0b900f104ce0 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:bb:cc:37:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:bb:cc:37:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:a3:83:ec:e1:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 15:47:10.812546 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.812544 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 15:47:10.812635 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.812614 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 15:47:10.815924 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.815899 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 15:47:10.816075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.815926 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 15:47:10.816863 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.816851 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 15:47:10.816916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.816867 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 15:47:10.816916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.816884 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:47:10.817712 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.817700 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:47:10.818862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.818850 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 15:47:10.820381 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820368 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:47:10.820502 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820491 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 15:47:10.820566 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820527 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 24 15:47:10.820566 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820540 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 15:47:10.820656 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820570 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 15:47:10.820656 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820583 2580 kubelet.go:398] "Adding apiserver pod source" Jul 24 15:47:10.820656 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820594 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 15:47:10.820971 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820957 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:47:10.821072 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.820984 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:47:10.823148 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.823135 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 15:47:10.824904 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.824888 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 15:47:10.824987 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.824926 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 15:47:10.827209 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827195 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 15:47:10.827277 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827217 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 15:47:10.827277 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827229 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 15:47:10.827277 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827238 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 15:47:10.827277 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827249 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 15:47:10.827277 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827258 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 15:47:10.827277 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827269 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 15:47:10.827277 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827276 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 15:47:10.827497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827287 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 15:47:10.827497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827296 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 15:47:10.827497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827312 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 15:47:10.827497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.827321 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 15:47:10.828111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.828100 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 15:47:10.828206 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.828113 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 15:47:10.829875 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.829845 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 15:47:10.829875 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.829844 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 15:47:10.831275 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.831260 2580 server.go:1268] "Started kubelet" Jul 24 15:47:10.831353 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.831331 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 15:47:10.831433 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.831341 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 15:47:10.831478 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.831461 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 15:47:10.832182 ip-10-0-140-155 systemd[1]: Started Kubernetes Kubelet. Jul 24 15:47:10.833748 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.833734 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 24 15:47:10.837541 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.837521 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 15:47:10.838387 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.838364 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p7jt2" Jul 24 15:47:10.838624 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.838602 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 15:47:10.840675 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.840655 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p7jt2" Jul 24 15:47:10.843464 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.843441 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 15:47:10.844284 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.843310 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-155.ec2.internal.18c54465ae700587 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-155.ec2.internal,UID:ip-10-0-140-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-155.ec2.internal,},FirstTimestamp:2026-07-24 15:47:10.831232391 +0000 UTC m=+0.389623102,LastTimestamp:2026-07-24 15:47:10.831232391 +0000 UTC m=+0.389623102,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-155.ec2.internal,}" Jul 24 15:47:10.845138 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.845119 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 15:47:10.845229 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.845138 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 15:47:10.845305 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.845287 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:10.846815 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.845729 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 15:47:10.846815 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.845746 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 15:47:10.846815 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.845850 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 24 15:47:10.846815 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.845857 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 24 15:47:10.846815 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.846535 2580 factory.go:55] Registering systemd factory Jul 24 15:47:10.846815 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.846552 2580 factory.go:223] Registration of the systemd container factory successfully Jul 24 15:47:10.847111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.846849 2580 factory.go:153] Registering CRI-O factory Jul 24 15:47:10.847111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.846861 2580 factory.go:223] Registration of the crio container factory successfully Jul 24 15:47:10.847111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.846948 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 15:47:10.847111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.846993 2580 factory.go:103] Registering Raw factory Jul 24 15:47:10.847111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.847011 2580 manager.go:1196] Started watching for new ooms in manager Jul 24 15:47:10.847317 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.847193 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 15:47:10.847579 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.847556 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:47:10.848088 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.848073 2580 manager.go:349] Starting recovery of all containers Jul 24 15:47:10.850692 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.850667 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-155.ec2.internal\" not found" node="ip-10-0-140-155.ec2.internal" Jul 24 15:47:10.857922 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.857786 2580 manager.go:354] Recovery completed Jul 24 15:47:10.861864 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.861850 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:47:10.864087 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.864071 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:47:10.864142 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.864103 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:47:10.864142 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.864118 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:47:10.864604 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.864588 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 15:47:10.864680 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.864604 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 15:47:10.864680 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.864618 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:47:10.864680 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.864622 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:47:10.865518 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.865505 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:47:10.865578 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.865529 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:47:10.865578 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.865538 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:47:10.866787 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.866772 2580 policy_none.go:49] "None policy: Start" Jul 24 15:47:10.866830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.866794 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 15:47:10.866830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.866807 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 15:47:10.867783 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.867763 2580 policy_none.go:47] "Start" Jul 24 15:47:10.907756 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.907737 2580 manager.go:337] "Starting Device Plugin manager" Jul 24 15:47:10.907828 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.907773 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 15:47:10.907828 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.907788 2580 server.go:87] "Starting device plugin registration server" Jul 24 15:47:10.908029 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.908011 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 15:47:10.908114 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.908025 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 15:47:10.908184 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.908133 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 15:47:10.908266 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.908254 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 15:47:10.908377 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.908267 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 15:47:10.908710 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.908688 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 15:47:10.908774 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.908743 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:10.965694 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.965667 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 15:47:10.966993 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.966972 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 15:47:10.967074 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.967006 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 15:47:10.967074 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.967023 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 15:47:10.967074 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:10.967050 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 15:47:10.969297 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:10.969279 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:47:11.008274 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.008218 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:47:11.009033 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.009018 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:47:11.009098 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.009047 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:47:11.009098 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.009058 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:47:11.009098 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.009079 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.016993 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.016979 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.017033 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.016998 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-155.ec2.internal\": node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.030530 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.030512 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.067432 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.067415 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal"] Jul 24 15:47:11.067482 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.067464 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:47:11.068776 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.068758 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:47:11.068825 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.068788 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:47:11.068825 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.068797 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:47:11.069869 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.069857 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:47:11.070003 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.069987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.070035 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.070021 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:47:11.070507 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.070493 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:47:11.070581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.070523 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:47:11.070581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.070537 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:47:11.070581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.070568 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:47:11.070666 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.070589 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:47:11.070666 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.070605 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:47:11.071640 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.071627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.071698 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.071655 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:47:11.072244 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.072227 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:47:11.072320 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.072252 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:47:11.072320 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.072262 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:47:11.087675 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.087657 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-155.ec2.internal\" not found" node="ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.092344 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.092331 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-155.ec2.internal\" not found" node="ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.130770 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.130754 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.231221 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.231202 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.247307 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.247287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b04f1b286f4eb31ede2f623a42cec623-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal\" (UID: \"b04f1b286f4eb31ede2f623a42cec623\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.247365 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.247314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b04f1b286f4eb31ede2f623a42cec623-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal\" (UID: \"b04f1b286f4eb31ede2f623a42cec623\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.247365 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.247330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0202295370f109491358a25fd64440ce-config\") pod \"kube-apiserver-proxy-ip-10-0-140-155.ec2.internal\" (UID: \"0202295370f109491358a25fd64440ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.331777 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.331704 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.348035 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.348011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b04f1b286f4eb31ede2f623a42cec623-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal\" (UID: \"b04f1b286f4eb31ede2f623a42cec623\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.348145 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.348041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0202295370f109491358a25fd64440ce-config\") pod \"kube-apiserver-proxy-ip-10-0-140-155.ec2.internal\" (UID: \"0202295370f109491358a25fd64440ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.348145 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.348058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b04f1b286f4eb31ede2f623a42cec623-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal\" (UID: \"b04f1b286f4eb31ede2f623a42cec623\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.348145 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.348103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b04f1b286f4eb31ede2f623a42cec623-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal\" (UID: \"b04f1b286f4eb31ede2f623a42cec623\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.348145 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.348109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b04f1b286f4eb31ede2f623a42cec623-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal\" (UID: \"b04f1b286f4eb31ede2f623a42cec623\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.348145 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.348125 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0202295370f109491358a25fd64440ce-config\") pod \"kube-apiserver-proxy-ip-10-0-140-155.ec2.internal\" (UID: \"0202295370f109491358a25fd64440ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.390179 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.390144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.395983 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.395970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" Jul 24 15:47:11.432011 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.431987 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.532483 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.532449 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.633024 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.632971 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.733516 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.733486 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.780067 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.780043 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 15:47:11.780543 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.780223 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:47:11.780543 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.780223 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:47:11.834538 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.834515 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.845270 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.845235 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:42:11 +0000 UTC" deadline="2028-02-14 13:53:01.813770461 +0000 UTC" Jul 24 15:47:11.845270 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.845263 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13678h5m49.96851166s" Jul 24 15:47:11.889253 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.889112 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:47:11.933665 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:11.933630 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0202295370f109491358a25fd64440ce.slice/crio-e53f8b8e8947eb0d6f3d6d68f6e094a73909f8463324ae40f4c9b93e67a177d9 WatchSource:0}: Error finding container e53f8b8e8947eb0d6f3d6d68f6e094a73909f8463324ae40f4c9b93e67a177d9: Status 404 returned error can't find the container with id e53f8b8e8947eb0d6f3d6d68f6e094a73909f8463324ae40f4c9b93e67a177d9 Jul 24 15:47:11.934288 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:11.934265 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04f1b286f4eb31ede2f623a42cec623.slice/crio-1ad68dc0485c95c43a8027298431122b89beba5e6503fc843a8453c69272af8f WatchSource:0}: Error finding container 1ad68dc0485c95c43a8027298431122b89beba5e6503fc843a8453c69272af8f: Status 404 returned error can't find the container with id 1ad68dc0485c95c43a8027298431122b89beba5e6503fc843a8453c69272af8f Jul 24 15:47:11.935342 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:11.935322 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-155.ec2.internal\" not found" Jul 24 15:47:11.939076 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.939055 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:47:11.970219 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.970179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" event={"ID":"b04f1b286f4eb31ede2f623a42cec623","Type":"ContainerStarted","Data":"1ad68dc0485c95c43a8027298431122b89beba5e6503fc843a8453c69272af8f"} Jul 24 15:47:11.971073 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.971052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" event={"ID":"0202295370f109491358a25fd64440ce","Type":"ContainerStarted","Data":"e53f8b8e8947eb0d6f3d6d68f6e094a73909f8463324ae40f4c9b93e67a177d9"} Jul 24 15:47:11.973068 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:11.973052 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:47:12.045543 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:12.045517 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" Jul 24 15:47:12.055720 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:12.055700 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:47:12.056575 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:12.056564 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" Jul 24 15:47:12.062395 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:12.062383 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:47:13.858521 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.858492 2580 apiserver.go:52] "Watching apiserver" Jul 24 15:47:13.864886 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.864858 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 15:47:13.865284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.865250 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-rb6dp","kube-system/global-pull-secret-syncer-txr7v","openshift-cluster-node-tuning-operator/tuned-hgxwj","openshift-dns/node-resolver-xmbr7","openshift-image-registry/node-ca-gfbz8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal","openshift-multus/multus-additional-cni-plugins-7mk9f","openshift-multus/multus-qswst","openshift-multus/network-metrics-daemon-ngh44","kube-system/konnectivity-agent-hphrq","kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58","openshift-network-diagnostics/network-check-target-6lqpv","openshift-network-operator/iptables-alerter-2pxsl"] Jul 24 15:47:13.868212 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.867921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.870812 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.870788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:13.870921 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:13.870891 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:13.871065 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.870987 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 15:47:13.871376 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.871239 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 15:47:13.871491 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.871390 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-srxxh\"" Jul 24 15:47:13.872214 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.871959 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 15:47:13.872214 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.871980 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 15:47:13.872214 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.872031 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 15:47:13.872998 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.872467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.874219 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.874197 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:13.874406 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.874386 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:13.874519 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.874425 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k9m9k\"" Jul 24 15:47:13.874961 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.874928 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.876612 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.876591 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 15:47:13.876835 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.876817 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w896k\"" Jul 24 15:47:13.877136 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.877119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 15:47:13.877341 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.877323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.878818 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.878803 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 15:47:13.879184 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.879169 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 15:47:13.879361 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.879205 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lm9gm\"" Jul 24 15:47:13.879461 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.879261 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 15:47:13.879596 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.879573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.881386 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.881202 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:13.881493 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.881453 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tbtg5\"" Jul 24 15:47:13.881564 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.881531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:13.881684 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.881663 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 15:47:13.881840 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.881811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:13.881980 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:13.881901 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:13.882163 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.882145 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 15:47:13.884793 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.884777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.887294 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.887303 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.887426 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.887531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.887712 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.887694 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s9928\"" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.887962 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 15:47:13.888193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.888186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:13.888579 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:13.888242 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:13.890648 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.890591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892418 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892522 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2shtm\"" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-os-release\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9s5\" (UniqueName: \"kubernetes.io/projected/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-kube-api-access-kf9s5\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-host\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cc5d671-a906-4ed2-a082-b73713af302b-iptables-alerter-script\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cc5d671-a906-4ed2-a082-b73713af302b-host-slash\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-kubelet\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-ovn\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-ovnkube-script-lib\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-systemd\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-lib-modules\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.892973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-systemd-units\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-log-socket\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.893286 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkd5\" (UniqueName: \"kubernetes.io/projected/b0008e3a-6e40-463c-b57a-5995805b848f-kube-api-access-9rkd5\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b687088-c49c-4cdb-a4bd-d0099456e00d-host\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-run-netns\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/382a7fe3-6e5c-455d-a354-16a1759ba389-tmp\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qswst" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-cni-bin\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-env-overrides\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893329 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysconfig\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-tuned\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xzh\" (UniqueName: \"kubernetes.io/projected/646c947d-c586-4522-9712-0204b10c4992-kube-api-access-c9xzh\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-slash\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-node-log\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cnibin\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-cni-netd\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-kubernetes\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-var-lib-kubelet\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.894292 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkd2\" (UniqueName: \"kubernetes.io/projected/3b687088-c49c-4cdb-a4bd-d0099456e00d-kube-api-access-bbkd2\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-ovnkube-config\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0008e3a-6e40-463c-b57a-5995805b848f-ovn-node-metrics-cert\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b687088-c49c-4cdb-a4bd-d0099456e00d-serviceca\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-var-lib-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-modprobe-d\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-system-cni-dir\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-systemd\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysctl-d\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.893992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysctl-conf\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-etc-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-run\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4qw\" (UniqueName: \"kubernetes.io/projected/382a7fe3-6e5c-455d-a354-16a1759ba389-kube-api-access-bp4qw\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.895101 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646c947d-c586-4522-9712-0204b10c4992-tmp-dir\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/646c947d-c586-4522-9712-0204b10c4992-hosts-file\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsjf6\" (UniqueName: \"kubernetes.io/projected/e0add84e-ed07-4e3e-99ac-29a994e6b55e-kube-api-access-hsjf6\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-sys\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8npf\" (UniqueName: \"kubernetes.io/projected/8cc5d671-a906-4ed2-a082-b73713af302b-kube-api-access-q8npf\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.894896 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.895192 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bspnx\"" Jul 24 15:47:13.896075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.895221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.896689 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.896655 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:47:13.896897 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.896878 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 15:47:13.897085 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.897061 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5z8kt\"" Jul 24 15:47:13.897362 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.897346 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 15:47:13.897436 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.897414 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 15:47:13.915123 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.915102 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rf8j9" Jul 24 15:47:13.924340 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.923983 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rf8j9" Jul 24 15:47:13.985616 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.985599 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 15:47:13.995409 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-kubelet\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.995507 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkd2\" (UniqueName: \"kubernetes.io/projected/3b687088-c49c-4cdb-a4bd-d0099456e00d-kube-api-access-bbkd2\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.995507 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-ovnkube-config\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.995507 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0008e3a-6e40-463c-b57a-5995805b848f-ovn-node-metrics-cert\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.995507 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.995507 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-os-release\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b687088-c49c-4cdb-a4bd-d0099456e00d-serviceca\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-var-lib-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-modprobe-d\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-system-cni-dir\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-system-cni-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b04e560a-d00a-46d8-82ee-e9ae357824ee-cni-binary-copy\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4tp\" (UniqueName: \"kubernetes.io/projected/b04e560a-d00a-46d8-82ee-e9ae357824ee-kube-api-access-kq4tp\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-systemd\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysctl-d\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysctl-conf\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-system-cni-dir\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8d08-5deb-4923-b672-a15772b5065c-konnectivity-ca\") pod \"konnectivity-agent-hphrq\" (UID: \"c6fe8d08-5deb-4923-b672-a15772b5065c\") " pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-daemon-config\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.995862 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-systemd\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-etc-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995837 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-run\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4qw\" (UniqueName: \"kubernetes.io/projected/382a7fe3-6e5c-455d-a354-16a1759ba389-kube-api-access-bp4qw\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-hostroot\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-socket-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysctl-conf\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-device-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646c947d-c586-4522-9712-0204b10c4992-tmp-dir\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.995971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysctl-d\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-ovnkube-config\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-netns\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.996718 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/646c947d-c586-4522-9712-0204b10c4992-hosts-file\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-run\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-var-lib-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-etc-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsjf6\" (UniqueName: \"kubernetes.io/projected/e0add84e-ed07-4e3e-99ac-29a994e6b55e-kube-api-access-hsjf6\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-modprobe-d\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6fe8d08-5deb-4923-b672-a15772b5065c-agent-certs\") pod \"konnectivity-agent-hphrq\" (UID: \"c6fe8d08-5deb-4923-b672-a15772b5065c\") " pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-k8s-cni-cncf-io\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b687088-c49c-4cdb-a4bd-d0099456e00d-serviceca\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3c5e7b6-0692-4501-883c-29b33100cbd9-kubelet-config\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-sys\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8npf\" (UniqueName: \"kubernetes.io/projected/8cc5d671-a906-4ed2-a082-b73713af302b-kube-api-access-q8npf\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-etc-kubernetes\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.997577 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3c5e7b6-0692-4501-883c-29b33100cbd9-dbus\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-openvswitch\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-os-release\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-sys\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646c947d-c586-4522-9712-0204b10c4992-tmp-dir\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/646c947d-c586-4522-9712-0204b10c4992-hosts-file\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9s5\" (UniqueName: \"kubernetes.io/projected/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-kube-api-access-kf9s5\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-host\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cc5d671-a906-4ed2-a082-b73713af302b-iptables-alerter-script\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-os-release\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-host\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cc5d671-a906-4ed2-a082-b73713af302b-host-slash\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-conf-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-kubelet\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-ovn\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cc5d671-a906-4ed2-a082-b73713af302b-host-slash\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.998339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-ovnkube-script-lib\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-run-ovn\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-systemd\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-kubelet\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-lib-modules\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-systemd\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-cni-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-lib-modules\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.996999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-socket-dir-parent\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-systemd-units\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-log-socket\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-log-socket\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997094 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-systemd-units\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-registration-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ggb\" (UniqueName: \"kubernetes.io/projected/212af0f7-aa4d-4203-b23b-30192592758e-kube-api-access-d7ggb\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.999060 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cc5d671-a906-4ed2-a082-b73713af302b-iptables-alerter-script\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkd5\" (UniqueName: \"kubernetes.io/projected/b0008e3a-6e40-463c-b57a-5995805b848f-kube-api-access-9rkd5\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-sys-fs\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b687088-c49c-4cdb-a4bd-d0099456e00d-host\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-run-netns\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/382a7fe3-6e5c-455d-a354-16a1759ba389-tmp\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-cnibin\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-multus-certs\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-cni-bin\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-ovnkube-script-lib\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-env-overrides\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysconfig\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-tuned\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997486 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b687088-c49c-4cdb-a4bd-d0099456e00d-host\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997489 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-run-netns\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-cni-bin\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:13.999764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-cni-bin\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-cni-multus\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-sysconfig\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xzh\" (UniqueName: \"kubernetes.io/projected/646c947d-c586-4522-9712-0204b10c4992-kube-api-access-c9xzh\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-slash\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-node-log\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cnibin\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-cni-netd\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-kubernetes\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-slash\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-var-lib-kubelet\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:13.997845 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-var-lib-kubelet\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0add84e-ed07-4e3e-99ac-29a994e6b55e-cnibin\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0008e3a-6e40-463c-b57a-5995805b848f-env-overrides\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-host-cni-netd\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.000661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-kubernetes\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.001541 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.997913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0008e3a-6e40-463c-b57a-5995805b848f-node-log\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.001541 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:13.997927 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:47:14.497890075 +0000 UTC m=+3.019498611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:14.001541 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:13.999720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0008e3a-6e40-463c-b57a-5995805b848f-ovn-node-metrics-cert\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.001541 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.000118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/382a7fe3-6e5c-455d-a354-16a1759ba389-tmp\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.001541 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.000167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/382a7fe3-6e5c-455d-a354-16a1759ba389-etc-tuned\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.005907 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.005863 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:47:14.005907 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.005907 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:47:14.006839 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.005921 2580 projected.go:196] Error preparing data for projected volume kube-api-access-256v8 for pod openshift-network-diagnostics/network-check-target-6lqpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:14.006839 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.006013 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8 podName:4c3aa307-3e72-44fb-bd78-b6ca76736bd2 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:14.50598055 +0000 UTC m=+3.027589069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-256v8" (UniqueName: "kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8") pod "network-check-target-6lqpv" (UID: "4c3aa307-3e72-44fb-bd78-b6ca76736bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:14.011016 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.010989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9s5\" (UniqueName: \"kubernetes.io/projected/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-kube-api-access-kf9s5\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:14.011016 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.010999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xzh\" (UniqueName: \"kubernetes.io/projected/646c947d-c586-4522-9712-0204b10c4992-kube-api-access-c9xzh\") pod \"node-resolver-xmbr7\" (UID: \"646c947d-c586-4522-9712-0204b10c4992\") " pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:14.012377 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.012232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4qw\" (UniqueName: \"kubernetes.io/projected/382a7fe3-6e5c-455d-a354-16a1759ba389-kube-api-access-bp4qw\") pod \"tuned-hgxwj\" (UID: \"382a7fe3-6e5c-455d-a354-16a1759ba389\") " pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.012764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.012731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkd2\" (UniqueName: \"kubernetes.io/projected/3b687088-c49c-4cdb-a4bd-d0099456e00d-kube-api-access-bbkd2\") pod \"node-ca-gfbz8\" (UID: \"3b687088-c49c-4cdb-a4bd-d0099456e00d\") " pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:14.012858 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.012820 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsjf6\" (UniqueName: \"kubernetes.io/projected/e0add84e-ed07-4e3e-99ac-29a994e6b55e-kube-api-access-hsjf6\") pod \"multus-additional-cni-plugins-7mk9f\" (UID: \"e0add84e-ed07-4e3e-99ac-29a994e6b55e\") " pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:14.012920 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.012901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8npf\" (UniqueName: \"kubernetes.io/projected/8cc5d671-a906-4ed2-a082-b73713af302b-kube-api-access-q8npf\") pod \"iptables-alerter-2pxsl\" (UID: \"8cc5d671-a906-4ed2-a082-b73713af302b\") " pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:14.014734 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.014712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkd5\" (UniqueName: \"kubernetes.io/projected/b0008e3a-6e40-463c-b57a-5995805b848f-kube-api-access-9rkd5\") pod \"ovnkube-node-rb6dp\" (UID: \"b0008e3a-6e40-463c-b57a-5995805b848f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.098253 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-conf-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-cni-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-socket-dir-parent\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-conf-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-registration-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.098418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.098418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ggb\" (UniqueName: \"kubernetes.io/projected/212af0f7-aa4d-4203-b23b-30192592758e-kube-api-access-d7ggb\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-sys-fs\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-registration-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-cnibin\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-multus-certs\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-cni-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-cni-bin\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-cni-bin\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-socket-dir-parent\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-sys-fs\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-cni-multus\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-cnibin\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-multus-certs\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-kubelet\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-cni-multus\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.098637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-os-release\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-system-cni-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b04e560a-d00a-46d8-82ee-e9ae357824ee-cni-binary-copy\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4tp\" (UniqueName: \"kubernetes.io/projected/b04e560a-d00a-46d8-82ee-e9ae357824ee-kube-api-access-kq4tp\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8d08-5deb-4923-b672-a15772b5065c-konnectivity-ca\") pod \"konnectivity-agent-hphrq\" (UID: \"c6fe8d08-5deb-4923-b672-a15772b5065c\") " pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-os-release\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-daemon-config\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-hostroot\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-socket-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-device-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-netns\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6fe8d08-5deb-4923-b672-a15772b5065c-agent-certs\") pod \"konnectivity-agent-hphrq\" (UID: \"c6fe8d08-5deb-4923-b672-a15772b5065c\") " pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-k8s-cni-cncf-io\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3c5e7b6-0692-4501-883c-29b33100cbd9-kubelet-config\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-system-cni-dir\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-etc-kubernetes\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099165 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3c5e7b6-0692-4501-883c-29b33100cbd9-dbus\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3c5e7b6-0692-4501-883c-29b33100cbd9-dbus\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.098715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-var-lib-kubelet\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.098682 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.099091 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret podName:b3c5e7b6-0692-4501-883c-29b33100cbd9 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:14.599078104 +0000 UTC m=+3.120686623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret") pod "global-pull-secret-syncer-txr7v" (UID: "b3c5e7b6-0692-4501-883c-29b33100cbd9") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-hostroot\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe8d08-5deb-4923-b672-a15772b5065c-konnectivity-ca\") pod \"konnectivity-agent-hphrq\" (UID: \"c6fe8d08-5deb-4923-b672-a15772b5065c\") " pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099174 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-k8s-cni-cncf-io\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-socket-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-host-run-netns\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/212af0f7-aa4d-4203-b23b-30192592758e-device-dir\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b04e560a-d00a-46d8-82ee-e9ae357824ee-etc-kubernetes\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b04e560a-d00a-46d8-82ee-e9ae357824ee-cni-binary-copy\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3c5e7b6-0692-4501-883c-29b33100cbd9-kubelet-config\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:14.099821 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.099584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b04e560a-d00a-46d8-82ee-e9ae357824ee-multus-daemon-config\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.101849 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.101830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6fe8d08-5deb-4923-b672-a15772b5065c-agent-certs\") pod \"konnectivity-agent-hphrq\" (UID: \"c6fe8d08-5deb-4923-b672-a15772b5065c\") " pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:14.110187 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.110169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4tp\" (UniqueName: \"kubernetes.io/projected/b04e560a-d00a-46d8-82ee-e9ae357824ee-kube-api-access-kq4tp\") pod \"multus-qswst\" (UID: \"b04e560a-d00a-46d8-82ee-e9ae357824ee\") " pod="openshift-multus/multus-qswst" Jul 24 15:47:14.110255 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.110187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ggb\" (UniqueName: \"kubernetes.io/projected/212af0f7-aa4d-4203-b23b-30192592758e-kube-api-access-d7ggb\") pod \"aws-ebs-csi-driver-node-n2h58\" (UID: \"212af0f7-aa4d-4203-b23b-30192592758e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.183215 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.183150 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" Jul 24 15:47:14.184786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.184770 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:47:14.192201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.192182 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" Jul 24 15:47:14.200790 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.200773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xmbr7" Jul 24 15:47:14.209732 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.209715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gfbz8" Jul 24 15:47:14.216618 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.216600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pxsl" Jul 24 15:47:14.224415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.224392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:14.231386 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.231368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:14.239014 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.238998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qswst" Jul 24 15:47:14.244870 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.244844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" Jul 24 15:47:14.416752 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.416721 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:47:14.502713 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.502641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:14.502836 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.502760 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:14.502836 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.502807 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:47:15.50279575 +0000 UTC m=+4.024404268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:14.603227 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.603197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:14.603410 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.603272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:14.603410 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.603365 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:14.603410 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.603386 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:47:14.603554 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.603415 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:47:14.603554 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.603429 2580 projected.go:196] Error preparing data for projected volume kube-api-access-256v8 for pod openshift-network-diagnostics/network-check-target-6lqpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:14.603554 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.603435 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret podName:b3c5e7b6-0692-4501-883c-29b33100cbd9 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:15.6034173 +0000 UTC m=+4.125025819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret") pod "global-pull-secret-syncer-txr7v" (UID: "b3c5e7b6-0692-4501-883c-29b33100cbd9") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:14.603554 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:14.603481 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8 podName:4c3aa307-3e72-44fb-bd78-b6ca76736bd2 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:15.603466434 +0000 UTC m=+4.125074956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-256v8" (UniqueName: "kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8") pod "network-check-target-6lqpv" (UID: "4c3aa307-3e72-44fb-bd78-b6ca76736bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:14.851274 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:14.851220 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc5d671_a906_4ed2_a082_b73713af302b.slice/crio-00bd438f5280f141a1eefbbfc61760fe1988735148db553964274eefef901aa4 WatchSource:0}: Error finding container 00bd438f5280f141a1eefbbfc61760fe1988735148db553964274eefef901aa4: Status 404 returned error can't find the container with id 00bd438f5280f141a1eefbbfc61760fe1988735148db553964274eefef901aa4 Jul 24 15:47:14.852806 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:14.852786 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fe8d08_5deb_4923_b672_a15772b5065c.slice/crio-ab0606a9dba7f534415217ad217acd0d780718626138760a09ca0ba8e6d58eb4 WatchSource:0}: Error finding container ab0606a9dba7f534415217ad217acd0d780718626138760a09ca0ba8e6d58eb4: Status 404 returned error can't find the container with id ab0606a9dba7f534415217ad217acd0d780718626138760a09ca0ba8e6d58eb4 Jul 24 15:47:14.856383 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:14.856361 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04e560a_d00a_46d8_82ee_e9ae357824ee.slice/crio-52099c92d4f1fb45647b9f134e761679b2a3c593dae6d64832ea9530ff7429aa WatchSource:0}: Error finding container 52099c92d4f1fb45647b9f134e761679b2a3c593dae6d64832ea9530ff7429aa: Status 404 returned error can't find the container with id 52099c92d4f1fb45647b9f134e761679b2a3c593dae6d64832ea9530ff7429aa Jul 24 15:47:14.856788 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:14.856766 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b687088_c49c_4cdb_a4bd_d0099456e00d.slice/crio-c053844d44644bbae945d8a79530f1e7c7b3f3789ee5cccb60f5026d228a1c6c WatchSource:0}: Error finding container c053844d44644bbae945d8a79530f1e7c7b3f3789ee5cccb60f5026d228a1c6c: Status 404 returned error can't find the container with id c053844d44644bbae945d8a79530f1e7c7b3f3789ee5cccb60f5026d228a1c6c Jul 24 15:47:14.859122 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:14.859041 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646c947d_c586_4522_9712_0204b10c4992.slice/crio-384a54e6f21115d89520178248938d3a992705ee7b4a4c252694cea0f9b7e2dc WatchSource:0}: Error finding container 384a54e6f21115d89520178248938d3a992705ee7b4a4c252694cea0f9b7e2dc: Status 404 returned error can't find the container with id 384a54e6f21115d89520178248938d3a992705ee7b4a4c252694cea0f9b7e2dc Jul 24 15:47:14.860391 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:14.860051 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0add84e_ed07_4e3e_99ac_29a994e6b55e.slice/crio-47fbad3ed03b837887a983a4e63a774fc49e0d0e68bf66454d3523a9085e7986 WatchSource:0}: Error finding container 47fbad3ed03b837887a983a4e63a774fc49e0d0e68bf66454d3523a9085e7986: Status 404 returned error can't find the container with id 47fbad3ed03b837887a983a4e63a774fc49e0d0e68bf66454d3523a9085e7986 Jul 24 15:47:14.927613 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.927478 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:42:13 +0000 UTC" deadline="2028-01-10 09:45:18.890630536 +0000 UTC" Jul 24 15:47:14.927700 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:14.927637 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12833h58m3.963013251s" Jul 24 15:47:15.004385 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.004355 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:15.004536 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.004449 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:15.004536 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.004504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:15.004647 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.004601 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:15.015916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.015884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" event={"ID":"382a7fe3-6e5c-455d-a354-16a1759ba389","Type":"ContainerStarted","Data":"1f54b8a396535498684ab9dda6365932da20aadf4455b7c7b2b59955fd14d553"} Jul 24 15:47:15.017033 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.017010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerStarted","Data":"47fbad3ed03b837887a983a4e63a774fc49e0d0e68bf66454d3523a9085e7986"} Jul 24 15:47:15.018187 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.018166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qswst" event={"ID":"b04e560a-d00a-46d8-82ee-e9ae357824ee","Type":"ContainerStarted","Data":"52099c92d4f1fb45647b9f134e761679b2a3c593dae6d64832ea9530ff7429aa"} Jul 24 15:47:15.019315 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.019292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" event={"ID":"212af0f7-aa4d-4203-b23b-30192592758e","Type":"ContainerStarted","Data":"78326c2dea2dfc3855aa59fb0a37a06c64a33869d0f0da2ba354e48accab9d61"} Jul 24 15:47:15.020584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.020559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gfbz8" event={"ID":"3b687088-c49c-4cdb-a4bd-d0099456e00d","Type":"ContainerStarted","Data":"c053844d44644bbae945d8a79530f1e7c7b3f3789ee5cccb60f5026d228a1c6c"} Jul 24 15:47:15.021978 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.021960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xmbr7" event={"ID":"646c947d-c586-4522-9712-0204b10c4992","Type":"ContainerStarted","Data":"384a54e6f21115d89520178248938d3a992705ee7b4a4c252694cea0f9b7e2dc"} Jul 24 15:47:15.023213 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.023192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hphrq" event={"ID":"c6fe8d08-5deb-4923-b672-a15772b5065c","Type":"ContainerStarted","Data":"ab0606a9dba7f534415217ad217acd0d780718626138760a09ca0ba8e6d58eb4"} Jul 24 15:47:15.024401 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.024377 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pxsl" event={"ID":"8cc5d671-a906-4ed2-a082-b73713af302b","Type":"ContainerStarted","Data":"00bd438f5280f141a1eefbbfc61760fe1988735148db553964274eefef901aa4"} Jul 24 15:47:15.026612 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.026578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" event={"ID":"0202295370f109491358a25fd64440ce","Type":"ContainerStarted","Data":"756bb49becdedd0efa0466e9f9e9ff9b4c3be89bb0d2bcf992e8a65446070551"} Jul 24 15:47:15.027854 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.027838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"f1ed71c25df64ffab07f8f4949e276d1402f5910f3a51befd4d872955be883a1"} Jul 24 15:47:15.044528 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.044492 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-155.ec2.internal" podStartSLOduration=2.044481407 podStartE2EDuration="2.044481407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:15.044223534 +0000 UTC m=+3.565832074" watchObservedRunningTime="2026-07-24 15:47:15.044481407 +0000 UTC m=+3.566089977" Jul 24 15:47:15.509960 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.509300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:15.509960 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.509468 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:15.509960 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.509535 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:47:17.509516076 +0000 UTC m=+6.031124594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:15.610470 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.610427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:15.610660 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.610513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:15.610660 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.610641 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:15.610783 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.610702 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret podName:b3c5e7b6-0692-4501-883c-29b33100cbd9 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:17.610684252 +0000 UTC m=+6.132292791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret") pod "global-pull-secret-syncer-txr7v" (UID: "b3c5e7b6-0692-4501-883c-29b33100cbd9") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:15.610845 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.610789 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:47:15.610845 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.610804 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:47:15.610845 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.610817 2580 projected.go:196] Error preparing data for projected volume kube-api-access-256v8 for pod openshift-network-diagnostics/network-check-target-6lqpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:15.611025 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:15.610855 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8 podName:4c3aa307-3e72-44fb-bd78-b6ca76736bd2 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:17.610842749 +0000 UTC m=+6.132451285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-256v8" (UniqueName: "kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8") pod "network-check-target-6lqpv" (UID: "4c3aa307-3e72-44fb-bd78-b6ca76736bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:15.928850 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.928734 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:42:13 +0000 UTC" deadline="2028-01-01 18:53:31.06511438 +0000 UTC" Jul 24 15:47:15.928850 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:15.928798 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12627h6m15.13632059s" Jul 24 15:47:16.006847 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:16.006811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:16.007044 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:16.006979 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:16.035540 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:16.035504 2580 generic.go:356] "Generic (PLEG): container finished" podID="b04f1b286f4eb31ede2f623a42cec623" containerID="421312cbe1c7a99bbdd90d4d68d0adbc86e30428be35750a4a5c974cbc147c50" exitCode=0 Jul 24 15:47:16.036575 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:16.036540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" event={"ID":"b04f1b286f4eb31ede2f623a42cec623","Type":"ContainerDied","Data":"421312cbe1c7a99bbdd90d4d68d0adbc86e30428be35750a4a5c974cbc147c50"} Jul 24 15:47:17.010961 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:17.004955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:17.010961 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:17.004970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:17.010961 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.005085 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:17.010961 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.005238 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:17.048406 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:17.048363 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" event={"ID":"b04f1b286f4eb31ede2f623a42cec623","Type":"ContainerStarted","Data":"b3f4ee781a744c3f1af2ee5c911acdf1cff8de03ee4bfd6fca18ac4d743c90f3"} Jul 24 15:47:17.529469 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:17.529433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:17.529656 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.529607 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:17.529741 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.529669 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:47:21.529650285 +0000 UTC m=+10.051258804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:17.629848 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:17.629800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:17.630071 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:17.629879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:17.630071 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.630061 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:47:17.630207 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.630082 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:47:17.630207 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.630096 2580 projected.go:196] Error preparing data for projected volume kube-api-access-256v8 for pod openshift-network-diagnostics/network-check-target-6lqpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:17.630207 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.630156 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8 podName:4c3aa307-3e72-44fb-bd78-b6ca76736bd2 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:21.630136633 +0000 UTC m=+10.151745150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-256v8" (UniqueName: "kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8") pod "network-check-target-6lqpv" (UID: "4c3aa307-3e72-44fb-bd78-b6ca76736bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:17.630560 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.630539 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:17.630657 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:17.630598 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret podName:b3c5e7b6-0692-4501-883c-29b33100cbd9 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:21.630581914 +0000 UTC m=+10.152190452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret") pod "global-pull-secret-syncer-txr7v" (UID: "b3c5e7b6-0692-4501-883c-29b33100cbd9") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:18.004733 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:18.004651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:18.004895 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:18.004801 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:19.005118 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:19.004569 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:19.005118 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:19.004688 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:19.005118 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:19.005009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:19.005118 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:19.005085 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:20.005589 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:20.005113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:20.005589 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:20.005251 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:21.004109 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:21.004035 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:21.004247 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.004179 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:21.004795 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:21.004767 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:21.004926 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.004873 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:21.565138 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:21.565096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:21.565547 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.565284 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:21.565547 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.565337 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:47:29.565323197 +0000 UTC m=+18.086931714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:21.665832 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:21.665797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:21.666007 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:21.665875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:21.666070 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.666050 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:47:21.666151 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.666074 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:47:21.666151 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.666089 2580 projected.go:196] Error preparing data for projected volume kube-api-access-256v8 for pod openshift-network-diagnostics/network-check-target-6lqpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:21.666151 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.666140 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8 podName:4c3aa307-3e72-44fb-bd78-b6ca76736bd2 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:29.666124701 +0000 UTC m=+18.187733219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-256v8" (UniqueName: "kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8") pod "network-check-target-6lqpv" (UID: "4c3aa307-3e72-44fb-bd78-b6ca76736bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:21.666574 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.666503 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:21.666574 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:21.666549 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret podName:b3c5e7b6-0692-4501-883c-29b33100cbd9 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:29.666534518 +0000 UTC m=+18.188143037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret") pod "global-pull-secret-syncer-txr7v" (UID: "b3c5e7b6-0692-4501-883c-29b33100cbd9") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:22.006496 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:22.005984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:22.006496 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:22.006108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:23.005750 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:23.005117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:23.005750 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:23.005243 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:23.005750 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:23.005330 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:23.005750 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:23.005411 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:24.006406 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:24.006371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:24.006824 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:24.006520 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:25.004218 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:25.004185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:25.004406 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:25.004185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:25.004406 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:25.004306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:25.004406 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:25.004391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:26.004116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:26.004080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:26.004590 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:26.004228 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:27.004104 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:27.004070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:27.004288 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:27.004178 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:27.004288 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:27.004239 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:27.004639 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:27.004349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:28.004499 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:28.004467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:28.004974 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:28.004592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:29.004604 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:29.004563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:29.004604 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:29.004593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:29.005175 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.004681 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:29.005175 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.004834 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:29.626719 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:29.626667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:29.626923 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.626836 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:29.626923 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.626907 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:47:45.626885624 +0000 UTC m=+34.148494147 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:29.727876 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:29.727833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:29.728092 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:29.727902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:29.728092 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.728020 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:29.728092 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.728062 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:47:29.728092 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.728082 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret podName:b3c5e7b6-0692-4501-883c-29b33100cbd9 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:45.728065425 +0000 UTC m=+34.249673943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret") pod "global-pull-secret-syncer-txr7v" (UID: "b3c5e7b6-0692-4501-883c-29b33100cbd9") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:29.728092 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.728088 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:47:29.728360 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.728105 2580 projected.go:196] Error preparing data for projected volume kube-api-access-256v8 for pod openshift-network-diagnostics/network-check-target-6lqpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:29.728360 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:29.728158 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8 podName:4c3aa307-3e72-44fb-bd78-b6ca76736bd2 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:45.728141724 +0000 UTC m=+34.249750244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-256v8" (UniqueName: "kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8") pod "network-check-target-6lqpv" (UID: "4c3aa307-3e72-44fb-bd78-b6ca76736bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:30.006557 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:30.006468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:30.007029 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:30.006588 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:31.004288 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:31.004246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:31.004288 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:31.004266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:31.004555 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:31.004371 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:31.004555 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:31.004528 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:32.005174 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:32.005142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:32.005587 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:32.005269 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:33.004297 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:33.004257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:33.004297 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:33.004278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:33.004532 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:33.004381 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:33.004532 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:33.004463 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:34.005029 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.004641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:34.005860 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:34.005077 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:34.084235 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.084202 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0add84e-ed07-4e3e-99ac-29a994e6b55e" containerID="aa8cf4c9d9efa691411e9198e5412a4f2c203c36c76bcbd96aed4aeeaf69a6b7" exitCode=0 Jul 24 15:47:34.084378 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.084240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerDied","Data":"aa8cf4c9d9efa691411e9198e5412a4f2c203c36c76bcbd96aed4aeeaf69a6b7"} Jul 24 15:47:34.086279 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.086255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qswst" event={"ID":"b04e560a-d00a-46d8-82ee-e9ae357824ee","Type":"ContainerStarted","Data":"017e3c26cc4b28465fbb172e381c0fae48d8b60a93e196feafb6c34c7ced08c2"} Jul 24 15:47:34.088304 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.088279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" event={"ID":"212af0f7-aa4d-4203-b23b-30192592758e","Type":"ContainerStarted","Data":"66927a8a926c4d729eebadbed01c934f6af27b607b2e13de7bf71b006a43f04f"} Jul 24 15:47:34.090393 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.090108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gfbz8" event={"ID":"3b687088-c49c-4cdb-a4bd-d0099456e00d","Type":"ContainerStarted","Data":"0cb1fd27898030a1e0624cd2007b545a04d6bf046cb5a955b66997d0f6705c48"} Jul 24 15:47:34.092348 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.092322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xmbr7" event={"ID":"646c947d-c586-4522-9712-0204b10c4992","Type":"ContainerStarted","Data":"9eb3a4fe5708acd15e3ffcdee7b4224efe4ce3d960a2ad969900747e6352c777"} Jul 24 15:47:34.095207 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.095181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hphrq" event={"ID":"c6fe8d08-5deb-4923-b672-a15772b5065c","Type":"ContainerStarted","Data":"788010552e51f31a8f883f681a6abe4974afe600f726f7f4f244a96b6826fe8c"} Jul 24 15:47:34.098337 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.098315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"958fc11008539a10907eefd25821b53e7036a263c68969a17b7136a6b71a50e2"} Jul 24 15:47:34.098439 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.098389 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"412df6761a418ad9de0cb1ba19821209f2f1389c04180ea58610cd679b8838f5"} Jul 24 15:47:34.098501 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.098443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"999c2c940e235d98d7ed316304a850d127952487c436b86c4416e6b9f98fa7c7"} Jul 24 15:47:34.098556 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.098498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"d7cb9a044ac009671a57962f243a8015125370aa2315ed6a66ac172fbbc6511b"} Jul 24 15:47:34.098556 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.098551 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"3f97b79eb27d339c175f761cafadd9ce3ce4b318457390b761bf2ac007b12232"} Jul 24 15:47:34.099861 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.099839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" event={"ID":"382a7fe3-6e5c-455d-a354-16a1759ba389","Type":"ContainerStarted","Data":"415dd66618c11ddefed05232dde4769f5d683aed069ff39f2ac6b51992e6c43f"} Jul 24 15:47:34.104513 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.103898 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-155.ec2.internal" podStartSLOduration=21.103886729 podStartE2EDuration="21.103886729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:17.064196737 +0000 UTC m=+5.585805277" watchObservedRunningTime="2026-07-24 15:47:34.103886729 +0000 UTC m=+22.625495268" Jul 24 15:47:34.115347 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.115303 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xmbr7" podStartSLOduration=3.533453315 podStartE2EDuration="22.115288446s" totalImagesPullingTime="18.581835131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:34.114723781 +0000 UTC m=+22.636332323" watchObservedRunningTime="2026-07-24 15:47:34.115288446 +0000 UTC m=+22.636896986" Jul 24 15:47:34.131997 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.131960 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hphrq" podStartSLOduration=11.404173951 podStartE2EDuration="22.131928338s" totalImagesPullingTime="10.727754387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:34.131421569 +0000 UTC m=+22.653030107" watchObservedRunningTime="2026-07-24 15:47:34.131928338 +0000 UTC m=+22.653536878" Jul 24 15:47:34.147328 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.147293 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qswst" podStartSLOduration=3.504961962 podStartE2EDuration="22.147279225s" totalImagesPullingTime="18.642317263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:34.146851231 +0000 UTC m=+22.668459788" watchObservedRunningTime="2026-07-24 15:47:34.147279225 +0000 UTC m=+22.668887766" Jul 24 15:47:34.176488 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.176435 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gfbz8" podStartSLOduration=3.592796287 podStartE2EDuration="22.176415963s" totalImagesPullingTime="18.583619676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:34.176341991 +0000 UTC m=+22.697950542" watchObservedRunningTime="2026-07-24 15:47:34.176415963 +0000 UTC m=+22.698024504" Jul 24 15:47:34.177066 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.177034 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hgxwj" podStartSLOduration=3.5942563119999997 podStartE2EDuration="22.177024732s" totalImagesPullingTime="18.58276842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:34.163226332 +0000 UTC m=+22.684834871" watchObservedRunningTime="2026-07-24 15:47:34.177024732 +0000 UTC m=+22.698633316" Jul 24 15:47:34.262707 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.262673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:34.263180 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.263162 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:34.907202 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.907167 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 15:47:34.959550 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.959429 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:47:34.907197221Z","UUID":"9712c622-d402-456c-9e7c-6026eef2ee93","Handler":null,"Name":"","Endpoint":""} Jul 24 15:47:34.961041 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.961021 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 15:47:34.961163 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:34.961068 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 15:47:35.004748 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:35.004669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:35.004925 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:35.004794 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:35.004925 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:35.004842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:35.005073 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:35.004925 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:35.106597 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:35.106554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"ff485b4fb8c016563d4d0e95cc0ee26c51ba70683d17f13b10e26c7ad88b1010"} Jul 24 15:47:35.108827 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:35.108793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" event={"ID":"212af0f7-aa4d-4203-b23b-30192592758e","Type":"ContainerStarted","Data":"eac57cf6667ff8726be88cba28555667ad0ea6ed3656e8334aa7b57317ee3cb3"} Jul 24 15:47:35.110557 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:35.110503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pxsl" event={"ID":"8cc5d671-a906-4ed2-a082-b73713af302b","Type":"ContainerStarted","Data":"149da8b6890d2695be61759d4f0dc618b8864c51c83ff7711619c15cf8a2cafc"} Jul 24 15:47:35.125598 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:35.125546 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2pxsl" podStartSLOduration=4.536517708 podStartE2EDuration="23.125530011s" totalImagesPullingTime="18.589012303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:35.124845127 +0000 UTC m=+23.646453670" watchObservedRunningTime="2026-07-24 15:47:35.125530011 +0000 UTC m=+23.647138551" Jul 24 15:47:36.006957 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:36.006910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:36.007206 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:36.007027 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:36.112443 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:36.112216 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 15:47:37.004641 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:37.004608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:37.004641 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:37.004632 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:37.004875 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:37.004720 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:37.004875 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:37.004855 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:37.120002 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:37.119967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"ffbdf7384039713dc105e26314cf321515561c49f3d3dce737c4f8d3f81eab7d"} Jul 24 15:47:37.121973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:37.121930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" event={"ID":"212af0f7-aa4d-4203-b23b-30192592758e","Type":"ContainerStarted","Data":"441ddd5d761de9e539b5a86d8871a78ccb05fc96b5099850c2d0c1a22624c571"} Jul 24 15:47:37.138768 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:37.138722 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2h58" podStartSLOduration=4.284911271 podStartE2EDuration="25.138708982s" totalImagesPullingTime="20.853797711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:37.138202777 +0000 UTC m=+25.659811317" watchObservedRunningTime="2026-07-24 15:47:37.138708982 +0000 UTC m=+25.660317553" Jul 24 15:47:38.004418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:38.004386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:38.004641 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:38.004522 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:39.004692 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.004513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:39.005272 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.004566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:39.005272 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:39.004771 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:39.005272 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:39.004839 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:39.129011 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.128979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" event={"ID":"b0008e3a-6e40-463c-b57a-5995805b848f","Type":"ContainerStarted","Data":"6081f7ccdddd59e20ed72ef5194e7693795363ec7190d682830ecb6ee1067f8e"} Jul 24 15:47:39.129261 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.129229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:39.130795 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.130774 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0add84e-ed07-4e3e-99ac-29a994e6b55e" containerID="2161b353365cadfedf3d7482a9ed53745ac90c6c5464d6d05806a8d6feaf34c5" exitCode=0 Jul 24 15:47:39.130900 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.130814 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerDied","Data":"2161b353365cadfedf3d7482a9ed53745ac90c6c5464d6d05806a8d6feaf34c5"} Jul 24 15:47:39.145841 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.145817 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:39.156796 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.156756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" podStartSLOduration=8.500654909 podStartE2EDuration="27.156742862s" totalImagesPullingTime="18.656087953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:39.156094801 +0000 UTC m=+27.677703340" watchObservedRunningTime="2026-07-24 15:47:39.156742862 +0000 UTC m=+27.678351402" Jul 24 15:47:39.414924 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.414885 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:39.415140 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.415051 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 15:47:39.415485 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:39.415468 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hphrq" Jul 24 15:47:40.004535 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.004504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:40.004755 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:40.004667 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:40.132973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.132928 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:40.133134 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.132999 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:40.148764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.148737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:47:40.701474 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.701230 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6lqpv"] Jul 24 15:47:40.701687 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.701588 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:40.701755 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:40.701701 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:40.704430 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.704402 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ngh44"] Jul 24 15:47:40.704561 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.704507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:40.704653 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:40.704625 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:40.705138 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.705115 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txr7v"] Jul 24 15:47:40.705232 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:40.705209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:40.705324 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:40.705304 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:41.137267 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:41.137235 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0add84e-ed07-4e3e-99ac-29a994e6b55e" containerID="35c208ca5611d4b99ae2d419d99d8b4f31901037c37401c0827acb151278b3c6" exitCode=0 Jul 24 15:47:41.137669 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:41.137323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerDied","Data":"35c208ca5611d4b99ae2d419d99d8b4f31901037c37401c0827acb151278b3c6"} Jul 24 15:47:42.005141 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:42.005117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:42.005233 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:42.005215 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:42.005326 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:42.005310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:42.005430 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:42.005412 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:42.141924 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:42.141893 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0add84e-ed07-4e3e-99ac-29a994e6b55e" containerID="23c81211187b5f00de54c611068300e3314eb6ca929055af00a4a668890be0e6" exitCode=0 Jul 24 15:47:42.142423 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:42.141977 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerDied","Data":"23c81211187b5f00de54c611068300e3314eb6ca929055af00a4a668890be0e6"} Jul 24 15:47:43.004182 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:43.004152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:43.004351 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:43.004268 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:44.005035 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:44.005001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:44.005639 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:44.005007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:44.005639 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:44.005132 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:44.005639 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:44.005209 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:45.004751 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:45.004665 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:45.004956 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.004893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txr7v" podUID="b3c5e7b6-0692-4501-883c-29b33100cbd9" Jul 24 15:47:45.643278 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:45.643241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:45.643726 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.643379 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:45.643726 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.643457 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:48:17.643434245 +0000 UTC m=+66.165042767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:47:45.744365 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:45.744323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:45.744546 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:45.744413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:45.744546 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.744478 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:45.744546 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.744544 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret podName:b3c5e7b6-0692-4501-883c-29b33100cbd9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:17.744528543 +0000 UTC m=+66.266137079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret") pod "global-pull-secret-syncer-txr7v" (UID: "b3c5e7b6-0692-4501-883c-29b33100cbd9") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:47:45.744673 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.744564 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:47:45.744673 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.744587 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:47:45.744673 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.744604 2580 projected.go:196] Error preparing data for projected volume kube-api-access-256v8 for pod openshift-network-diagnostics/network-check-target-6lqpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:45.744673 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:45.744646 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8 podName:4c3aa307-3e72-44fb-bd78-b6ca76736bd2 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:17.744633393 +0000 UTC m=+66.266241911 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-256v8" (UniqueName: "kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8") pod "network-check-target-6lqpv" (UID: "4c3aa307-3e72-44fb-bd78-b6ca76736bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:47:46.005189 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.004725 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:46.005189 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:46.004849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6lqpv" podUID="4c3aa307-3e72-44fb-bd78-b6ca76736bd2" Jul 24 15:47:46.005189 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.004915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:46.005189 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:46.005057 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngh44" podUID="1837b9cd-372b-4e0e-ab2a-ff6013c26aea" Jul 24 15:47:46.339836 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.339803 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-155.ec2.internal" event="NodeReady" Jul 24 15:47:46.340023 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.339994 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 15:47:46.383062 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.383029 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fw7z6"] Jul 24 15:47:46.417055 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.417026 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-srv29"] Jul 24 15:47:46.417195 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.417161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.419188 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.419153 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 15:47:46.419392 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.419378 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 15:47:46.419469 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.419427 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59fhg\"" Jul 24 15:47:46.439153 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.439135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fw7z6"] Jul 24 15:47:46.439252 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.439165 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srv29"] Jul 24 15:47:46.439252 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.439234 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.441350 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.441323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 15:47:46.441443 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.441374 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 15:47:46.441517 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.441501 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 15:47:46.441573 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.441558 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c9gpx\"" Jul 24 15:47:46.549378 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.549344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e0c010d-8dda-4460-89d2-4fcf81149a91-cert\") pod \"ingress-canary-srv29\" (UID: \"8e0c010d-8dda-4460-89d2-4fcf81149a91\") " pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.549378 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.549378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.549595 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.549396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f27ac16-5144-4a65-9a33-791650b37e8c-tmp-dir\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.549595 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.549459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7c2j\" (UniqueName: \"kubernetes.io/projected/2f27ac16-5144-4a65-9a33-791650b37e8c-kube-api-access-j7c2j\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.549595 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.549526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqnw\" (UniqueName: \"kubernetes.io/projected/8e0c010d-8dda-4460-89d2-4fcf81149a91-kube-api-access-fpqnw\") pod \"ingress-canary-srv29\" (UID: \"8e0c010d-8dda-4460-89d2-4fcf81149a91\") " pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.549595 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.549557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f27ac16-5144-4a65-9a33-791650b37e8c-config-volume\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.650661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.650581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7c2j\" (UniqueName: \"kubernetes.io/projected/2f27ac16-5144-4a65-9a33-791650b37e8c-kube-api-access-j7c2j\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.650661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.650619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqnw\" (UniqueName: \"kubernetes.io/projected/8e0c010d-8dda-4460-89d2-4fcf81149a91-kube-api-access-fpqnw\") pod \"ingress-canary-srv29\" (UID: \"8e0c010d-8dda-4460-89d2-4fcf81149a91\") " pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.650661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.650645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f27ac16-5144-4a65-9a33-791650b37e8c-config-volume\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.651205 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.650708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e0c010d-8dda-4460-89d2-4fcf81149a91-cert\") pod \"ingress-canary-srv29\" (UID: \"8e0c010d-8dda-4460-89d2-4fcf81149a91\") " pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.651205 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.650733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.651205 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.650756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f27ac16-5144-4a65-9a33-791650b37e8c-tmp-dir\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.651205 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:46.650840 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:47:46.651205 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:46.650913 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls podName:2f27ac16-5144-4a65-9a33-791650b37e8c nodeName:}" failed. No retries permitted until 2026-07-24 15:47:47.150892282 +0000 UTC m=+35.672500803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls") pod "dns-default-fw7z6" (UID: "2f27ac16-5144-4a65-9a33-791650b37e8c") : secret "dns-default-metrics-tls" not found Jul 24 15:47:46.651205 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.651126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f27ac16-5144-4a65-9a33-791650b37e8c-tmp-dir\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.651380 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.651311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f27ac16-5144-4a65-9a33-791650b37e8c-config-volume\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.655252 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.655102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e0c010d-8dda-4460-89d2-4fcf81149a91-cert\") pod \"ingress-canary-srv29\" (UID: \"8e0c010d-8dda-4460-89d2-4fcf81149a91\") " pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.658621 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.658600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7c2j\" (UniqueName: \"kubernetes.io/projected/2f27ac16-5144-4a65-9a33-791650b37e8c-kube-api-access-j7c2j\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:46.658870 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.658854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqnw\" (UniqueName: \"kubernetes.io/projected/8e0c010d-8dda-4460-89d2-4fcf81149a91-kube-api-access-fpqnw\") pod \"ingress-canary-srv29\" (UID: \"8e0c010d-8dda-4460-89d2-4fcf81149a91\") " pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.746908 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.746873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srv29" Jul 24 15:47:46.979281 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:46.979247 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srv29"] Jul 24 15:47:46.984407 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:46.984381 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0c010d_8dda_4460_89d2_4fcf81149a91.slice/crio-7f5234a3dad714d469449c8a7280bd6e9d17673e6f8ddc972c9a0f0a237190cd WatchSource:0}: Error finding container 7f5234a3dad714d469449c8a7280bd6e9d17673e6f8ddc972c9a0f0a237190cd: Status 404 returned error can't find the container with id 7f5234a3dad714d469449c8a7280bd6e9d17673e6f8ddc972c9a0f0a237190cd Jul 24 15:47:47.004470 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:47.004445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:47:47.006635 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:47.006616 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:47:47.153967 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:47.153916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srv29" event={"ID":"8e0c010d-8dda-4460-89d2-4fcf81149a91","Type":"ContainerStarted","Data":"7f5234a3dad714d469449c8a7280bd6e9d17673e6f8ddc972c9a0f0a237190cd"} Jul 24 15:47:47.154379 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:47.154351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:47.154487 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:47.154463 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:47:47.154546 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:47.154526 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls podName:2f27ac16-5144-4a65-9a33-791650b37e8c nodeName:}" failed. No retries permitted until 2026-07-24 15:47:48.154506272 +0000 UTC m=+36.676114804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls") pod "dns-default-fw7z6" (UID: "2f27ac16-5144-4a65-9a33-791650b37e8c") : secret "dns-default-metrics-tls" not found Jul 24 15:47:47.156644 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:47.156625 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0add84e-ed07-4e3e-99ac-29a994e6b55e" containerID="5a0ebfd07f6fa1929dcf2c2656dea332e055dfc6dc18ac5ce3123464d03b51e0" exitCode=0 Jul 24 15:47:47.156739 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:47.156665 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerDied","Data":"5a0ebfd07f6fa1929dcf2c2656dea332e055dfc6dc18ac5ce3123464d03b51e0"} Jul 24 15:47:48.004150 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.004076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:47:48.004150 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.004076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:47:48.006998 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.006978 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:47:48.007121 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.007002 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:47:48.007121 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.006978 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w7pwm\"" Jul 24 15:47:48.007121 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.007099 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k45mm\"" Jul 24 15:47:48.007269 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.007229 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:47:48.161539 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.161504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:48.161725 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:48.161669 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:47:48.161790 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:48.161747 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls podName:2f27ac16-5144-4a65-9a33-791650b37e8c nodeName:}" failed. No retries permitted until 2026-07-24 15:47:50.161725465 +0000 UTC m=+38.683333986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls") pod "dns-default-fw7z6" (UID: "2f27ac16-5144-4a65-9a33-791650b37e8c") : secret "dns-default-metrics-tls" not found Jul 24 15:47:48.165499 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.163772 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0add84e-ed07-4e3e-99ac-29a994e6b55e" containerID="a9bd67a6c4d93d871ad982e37ffecfed66b6cd328f890692324c9087a4a7c21e" exitCode=0 Jul 24 15:47:48.165499 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:48.163837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerDied","Data":"a9bd67a6c4d93d871ad982e37ffecfed66b6cd328f890692324c9087a4a7c21e"} Jul 24 15:47:49.171234 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:49.171012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" event={"ID":"e0add84e-ed07-4e3e-99ac-29a994e6b55e","Type":"ContainerStarted","Data":"28d7e62e41c7fe75a9d049e06ac825b30998374670a53bab34282f87b6017d95"} Jul 24 15:47:49.194867 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:49.194051 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7mk9f" podStartSLOduration=7.28731456 podStartE2EDuration="37.194029663s" totalImagesPullingTime="29.906715103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:49.193436091 +0000 UTC m=+37.715044642" watchObservedRunningTime="2026-07-24 15:47:49.194029663 +0000 UTC m=+37.715638204" Jul 24 15:47:50.174586 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:50.174543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srv29" event={"ID":"8e0c010d-8dda-4460-89d2-4fcf81149a91","Type":"ContainerStarted","Data":"196e4b61702f2ea8fc6af67244aed3e99e98ecb1c906791f9b2f991aa5b10b7c"} Jul 24 15:47:50.177712 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:50.177695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:50.177812 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:50.177801 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:47:50.177857 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:50.177847 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls podName:2f27ac16-5144-4a65-9a33-791650b37e8c nodeName:}" failed. No retries permitted until 2026-07-24 15:47:54.177835401 +0000 UTC m=+42.699443919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls") pod "dns-default-fw7z6" (UID: "2f27ac16-5144-4a65-9a33-791650b37e8c") : secret "dns-default-metrics-tls" not found Jul 24 15:47:50.190838 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:50.190795 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-srv29" podStartSLOduration=1.78555267 podStartE2EDuration="4.190781267s" totalImagesPullingTime="2.405228597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:50.1898985 +0000 UTC m=+38.711507039" watchObservedRunningTime="2026-07-24 15:47:50.190781267 +0000 UTC m=+38.712389848" Jul 24 15:47:54.205682 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:54.205648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:47:54.206106 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:54.205796 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:47:54.206106 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:54.205853 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls podName:2f27ac16-5144-4a65-9a33-791650b37e8c nodeName:}" failed. No retries permitted until 2026-07-24 15:48:02.20583948 +0000 UTC m=+50.727447997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls") pod "dns-default-fw7z6" (UID: "2f27ac16-5144-4a65-9a33-791650b37e8c") : secret "dns-default-metrics-tls" not found Jul 24 15:47:57.232358 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.232328 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s"] Jul 24 15:47:57.263040 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.263009 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s"] Jul 24 15:47:57.263175 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.263111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" Jul 24 15:47:57.265485 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.265456 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:57.265624 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.265499 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:57.265624 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.265593 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-975d7\"" Jul 24 15:47:57.333152 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.333126 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vzvhl"] Jul 24 15:47:57.356138 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.356118 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq"] Jul 24 15:47:57.356278 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.356245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.358094 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.358076 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fcl4d\"" Jul 24 15:47:57.358304 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.358292 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 15:47:57.358356 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.358326 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 15:47:57.358538 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.358526 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:57.358607 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.358594 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:57.363561 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.363545 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 15:47:57.373442 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.373423 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld"] Jul 24 15:47:57.373546 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.373529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.375271 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.375242 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 15:47:57.375584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.375564 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:57.375694 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.375566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 15:47:57.375694 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.375588 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tr6tt\"" Jul 24 15:47:57.376008 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.375990 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:57.389392 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.389373 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vzvhl"] Jul 24 15:47:57.389482 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.389455 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq"] Jul 24 15:47:57.389526 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.389500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.389573 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.389530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld"] Jul 24 15:47:57.391329 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.391312 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 15:47:57.391329 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.391323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pc79z\"" Jul 24 15:47:57.391474 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.391386 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:57.391474 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.391388 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 15:47:57.391634 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.391620 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:57.429273 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.429245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zbj\" (UniqueName: \"kubernetes.io/projected/aa78a585-835c-49d6-85b5-caeae5da578c-kube-api-access-29zbj\") pod \"volume-data-source-validator-549c6d54cf-flr6s\" (UID: \"aa78a585-835c-49d6-85b5-caeae5da578c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" Jul 24 15:47:57.444220 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.444199 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq"] Jul 24 15:47:57.469591 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.469565 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cd86f67cd-56fck"] Jul 24 15:47:57.469722 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.469684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:57.471601 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.471578 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 15:47:57.471711 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.471610 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l9bfq\"" Jul 24 15:47:57.471971 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.471958 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:57.472127 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.472114 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:57.493698 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.493640 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq"] Jul 24 15:47:57.493698 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.493673 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cd86f67cd-56fck"] Jul 24 15:47:57.493809 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.493753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.495481 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.495458 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 15:47:57.495648 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.495635 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 15:47:57.496131 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.496115 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 15:47:57.496278 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.496264 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-phckq\"" Jul 24 15:47:57.503700 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.503668 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 15:47:57.530013 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.529985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccvl\" (UniqueName: \"kubernetes.io/projected/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-kube-api-access-qccvl\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.530116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zbj\" (UniqueName: \"kubernetes.io/projected/aa78a585-835c-49d6-85b5-caeae5da578c-kube-api-access-29zbj\") pod \"volume-data-source-validator-549c6d54cf-flr6s\" (UID: \"aa78a585-835c-49d6-85b5-caeae5da578c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" Jul 24 15:47:57.530116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqkr\" (UniqueName: \"kubernetes.io/projected/563706c0-a505-4373-b987-923b16eb34da-kube-api-access-pzqkr\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.530116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b78c\" (UniqueName: \"kubernetes.io/projected/a4980202-43a2-43ea-a4db-90e02a50ff32-kube-api-access-9b78c\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.530116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-serving-cert\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.530250 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/563706c0-a505-4373-b987-923b16eb34da-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.530250 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/563706c0-a505-4373-b987-923b16eb34da-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.530250 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4980202-43a2-43ea-a4db-90e02a50ff32-config\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.530250 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-trusted-ca\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.530365 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-config\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.530365 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.530299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4980202-43a2-43ea-a4db-90e02a50ff32-serving-cert\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.538882 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.538862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zbj\" (UniqueName: \"kubernetes.io/projected/aa78a585-835c-49d6-85b5-caeae5da578c-kube-api-access-29zbj\") pod \"volume-data-source-validator-549c6d54cf-flr6s\" (UID: \"aa78a585-835c-49d6-85b5-caeae5da578c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" Jul 24 15:47:57.573844 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.573811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" Jul 24 15:47:57.630744 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.630708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-trusted-ca\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.630869 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.630754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klq67\" (UniqueName: \"kubernetes.io/projected/dc6432ef-0508-4d51-aba7-9331a7199df3-kube-api-access-klq67\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:57.630869 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.630777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3685788b-ec3f-4d48-9bf8-132acf4b90bf-ca-trust-extracted\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.630968 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.630855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqkr\" (UniqueName: \"kubernetes.io/projected/563706c0-a505-4373-b987-923b16eb34da-kube-api-access-pzqkr\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.630968 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.630926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-bound-sa-token\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.631116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.630990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-serving-cert\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.631116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/563706c0-a505-4373-b987-923b16eb34da-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.631116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-certificates\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.631116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-installation-pull-secrets\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.631283 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/563706c0-a505-4373-b987-923b16eb34da-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.631283 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-config\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.631283 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4980202-43a2-43ea-a4db-90e02a50ff32-serving-cert\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.631424 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccvl\" (UniqueName: \"kubernetes.io/projected/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-kube-api-access-qccvl\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.631424 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.631532 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b78c\" (UniqueName: \"kubernetes.io/projected/a4980202-43a2-43ea-a4db-90e02a50ff32-kube-api-access-9b78c\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.631532 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-image-registry-private-configuration\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.631631 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-trusted-ca\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.631631 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:57.631631 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4980202-43a2-43ea-a4db-90e02a50ff32-config\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.631783 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vs94\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-kube-api-access-9vs94\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.631924 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.631898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/563706c0-a505-4373-b987-923b16eb34da-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.632058 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.632031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-trusted-ca\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.632123 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.632064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-config\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.632491 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.632471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4980202-43a2-43ea-a4db-90e02a50ff32-config\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.633878 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.633855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/563706c0-a505-4373-b987-923b16eb34da-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.634050 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.634032 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4980202-43a2-43ea-a4db-90e02a50ff32-serving-cert\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.634176 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.634158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-serving-cert\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.638517 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.638476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqkr\" (UniqueName: \"kubernetes.io/projected/563706c0-a505-4373-b987-923b16eb34da-kube-api-access-pzqkr\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq\" (UID: \"563706c0-a505-4373-b987-923b16eb34da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.638776 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.638746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccvl\" (UniqueName: \"kubernetes.io/projected/f014c88e-fe9e-4ed5-9579-9c5f721bc6ff-kube-api-access-qccvl\") pod \"console-operator-584dcb6b54-vzvhl\" (UID: \"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.639253 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.639232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b78c\" (UniqueName: \"kubernetes.io/projected/a4980202-43a2-43ea-a4db-90e02a50ff32-kube-api-access-9b78c\") pod \"service-ca-operator-5d6c75f84f-8bcld\" (UID: \"a4980202-43a2-43ea-a4db-90e02a50ff32\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.666399 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.666174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:47:57.682250 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.682191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" Jul 24 15:47:57.697331 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.697308 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" Jul 24 15:47:57.726489 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.726463 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s"] Jul 24 15:47:57.732637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.732637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-image-registry-private-configuration\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.732637 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-trusted-ca\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vs94\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-kube-api-access-9vs94\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klq67\" (UniqueName: \"kubernetes.io/projected/dc6432ef-0508-4d51-aba7-9331a7199df3-kube-api-access-klq67\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:57.732718 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:57.732737 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd86f67cd-56fck: secret "image-registry-tls" not found Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3685788b-ec3f-4d48-9bf8-132acf4b90bf-ca-trust-extracted\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:57.732810 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls podName:3685788b-ec3f-4d48-9bf8-132acf4b90bf nodeName:}" failed. No retries permitted until 2026-07-24 15:47:58.232776157 +0000 UTC m=+46.754384677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls") pod "image-registry-cd86f67cd-56fck" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf") : secret "image-registry-tls" not found Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-bound-sa-token\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-certificates\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.732912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-installation-pull-secrets\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:57.733071 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.733093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3685788b-ec3f-4d48-9bf8-132acf4b90bf-ca-trust-extracted\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.733547 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:57.733489 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls podName:dc6432ef-0508-4d51-aba7-9331a7199df3 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:58.23347 +0000 UTC m=+46.755078540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls") pod "cluster-samples-operator-6748489656-nbbnq" (UID: "dc6432ef-0508-4d51-aba7-9331a7199df3") : secret "samples-operator-tls" not found Jul 24 15:47:57.734358 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.734074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-certificates\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.734581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.734519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-trusted-ca\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.734581 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:57.734537 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa78a585_835c_49d6_85b5_caeae5da578c.slice/crio-26240d822a81db6c0bf156654e155542099c0991891ba328ebedfee93c231a02 WatchSource:0}: Error finding container 26240d822a81db6c0bf156654e155542099c0991891ba328ebedfee93c231a02: Status 404 returned error can't find the container with id 26240d822a81db6c0bf156654e155542099c0991891ba328ebedfee93c231a02 Jul 24 15:47:57.737357 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.737338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-installation-pull-secrets\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.737432 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.737396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-image-registry-private-configuration\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.744613 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.744553 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-bound-sa-token\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.744709 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.744644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vs94\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-kube-api-access-9vs94\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:57.744775 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.744697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klq67\" (UniqueName: \"kubernetes.io/projected/dc6432ef-0508-4d51-aba7-9331a7199df3-kube-api-access-klq67\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:57.858465 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.858145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vzvhl"] Jul 24 15:47:57.859614 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:57.858674 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf014c88e_fe9e_4ed5_9579_9c5f721bc6ff.slice/crio-55be1d4c5cdbf82d2c7787b8936489c00f45260f08bda472fabc45d0db249eef WatchSource:0}: Error finding container 55be1d4c5cdbf82d2c7787b8936489c00f45260f08bda472fabc45d0db249eef: Status 404 returned error can't find the container with id 55be1d4c5cdbf82d2c7787b8936489c00f45260f08bda472fabc45d0db249eef Jul 24 15:47:57.871401 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.871378 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq"] Jul 24 15:47:57.874213 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:57.874187 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563706c0_a505_4373_b987_923b16eb34da.slice/crio-6216dab999ea6fab19e6a08c5deeaa7bd72d23b86cf4b752b39810b5cc9922c8 WatchSource:0}: Error finding container 6216dab999ea6fab19e6a08c5deeaa7bd72d23b86cf4b752b39810b5cc9922c8: Status 404 returned error can't find the container with id 6216dab999ea6fab19e6a08c5deeaa7bd72d23b86cf4b752b39810b5cc9922c8 Jul 24 15:47:57.888727 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:57.888698 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld"] Jul 24 15:47:57.892526 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:47:57.892503 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4980202_43a2_43ea_a4db_90e02a50ff32.slice/crio-ba05c62644a1d5fa8f7aba889cba197977a176204ea495b74c1dcc6d5c474e9b WatchSource:0}: Error finding container ba05c62644a1d5fa8f7aba889cba197977a176204ea495b74c1dcc6d5c474e9b: Status 404 returned error can't find the container with id ba05c62644a1d5fa8f7aba889cba197977a176204ea495b74c1dcc6d5c474e9b Jul 24 15:47:58.195464 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:58.195424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" event={"ID":"a4980202-43a2-43ea-a4db-90e02a50ff32","Type":"ContainerStarted","Data":"ba05c62644a1d5fa8f7aba889cba197977a176204ea495b74c1dcc6d5c474e9b"} Jul 24 15:47:58.196484 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:58.196439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" event={"ID":"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff","Type":"ContainerStarted","Data":"55be1d4c5cdbf82d2c7787b8936489c00f45260f08bda472fabc45d0db249eef"} Jul 24 15:47:58.197574 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:58.197551 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" event={"ID":"aa78a585-835c-49d6-85b5-caeae5da578c","Type":"ContainerStarted","Data":"26240d822a81db6c0bf156654e155542099c0991891ba328ebedfee93c231a02"} Jul 24 15:47:58.198585 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:58.198566 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" event={"ID":"563706c0-a505-4373-b987-923b16eb34da","Type":"ContainerStarted","Data":"6216dab999ea6fab19e6a08c5deeaa7bd72d23b86cf4b752b39810b5cc9922c8"} Jul 24 15:47:58.236207 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:58.236185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:58.236506 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:58.236215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:58.236506 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:58.236319 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:47:58.236506 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:58.236335 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd86f67cd-56fck: secret "image-registry-tls" not found Jul 24 15:47:58.236506 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:58.236344 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:47:58.236506 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:58.236382 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls podName:3685788b-ec3f-4d48-9bf8-132acf4b90bf nodeName:}" failed. No retries permitted until 2026-07-24 15:47:59.236367598 +0000 UTC m=+47.757976116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls") pod "image-registry-cd86f67cd-56fck" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf") : secret "image-registry-tls" not found Jul 24 15:47:58.236506 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:58.236394 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls podName:dc6432ef-0508-4d51-aba7-9331a7199df3 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:59.23638871 +0000 UTC m=+47.757997228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls") pod "cluster-samples-operator-6748489656-nbbnq" (UID: "dc6432ef-0508-4d51-aba7-9331a7199df3") : secret "samples-operator-tls" not found Jul 24 15:47:59.245066 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:59.245024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:47:59.245457 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:47:59.245093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:47:59.245457 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:59.245212 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:47:59.245457 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:59.245237 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd86f67cd-56fck: secret "image-registry-tls" not found Jul 24 15:47:59.245457 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:59.245287 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:47:59.245457 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:59.245302 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls podName:3685788b-ec3f-4d48-9bf8-132acf4b90bf nodeName:}" failed. No retries permitted until 2026-07-24 15:48:01.245279754 +0000 UTC m=+49.766888278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls") pod "image-registry-cd86f67cd-56fck" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf") : secret "image-registry-tls" not found Jul 24 15:47:59.245457 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:47:59.245333 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls podName:dc6432ef-0508-4d51-aba7-9331a7199df3 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:01.245320435 +0000 UTC m=+49.766928954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls") pod "cluster-samples-operator-6748489656-nbbnq" (UID: "dc6432ef-0508-4d51-aba7-9331a7199df3") : secret "samples-operator-tls" not found Jul 24 15:48:00.889245 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.889215 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mvvwq"] Jul 24 15:48:00.901756 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.901730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:00.907739 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.907715 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 15:48:00.911131 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.911112 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qb9zg\"" Jul 24 15:48:00.912063 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.912049 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 15:48:00.918903 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.918883 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 15:48:00.923304 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.923284 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 15:48:00.924490 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:00.924457 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvvwq"] Jul 24 15:48:01.060202 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.060158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgm8\" (UniqueName: \"kubernetes.io/projected/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-api-access-mbgm8\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.060390 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.060240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.060390 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.060284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.060390 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.060309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99e0f12c-caff-4e63-a132-ed8cea3448b9-crio-socket\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.060390 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.060332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99e0f12c-caff-4e63-a132-ed8cea3448b9-data-volume\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.161826 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.161739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgm8\" (UniqueName: \"kubernetes.io/projected/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-api-access-mbgm8\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.162005 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.161833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.162005 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.161878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.162005 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.161905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99e0f12c-caff-4e63-a132-ed8cea3448b9-crio-socket\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.162005 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.161985 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:48:01.162005 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.161992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99e0f12c-caff-4e63-a132-ed8cea3448b9-data-volume\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.162236 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.162049 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:01.662031475 +0000 UTC m=+50.183639996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : configmap "kube-rbac-proxy" not found Jul 24 15:48:01.162236 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.162054 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:48:01.162236 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.162114 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:01.662094945 +0000 UTC m=+50.183703462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : secret "insights-runtime-extractor-tls" not found Jul 24 15:48:01.162236 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.162215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99e0f12c-caff-4e63-a132-ed8cea3448b9-crio-socket\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.162406 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.162314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99e0f12c-caff-4e63-a132-ed8cea3448b9-data-volume\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.170578 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.170541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgm8\" (UniqueName: \"kubernetes.io/projected/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-api-access-mbgm8\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.263125 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.263092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:48:01.263295 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.263139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:48:01.263295 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.263239 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:48:01.263295 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.263262 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:48:01.263295 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.263283 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd86f67cd-56fck: secret "image-registry-tls" not found Jul 24 15:48:01.263449 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.263316 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls podName:dc6432ef-0508-4d51-aba7-9331a7199df3 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:05.263297456 +0000 UTC m=+53.784905986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls") pod "cluster-samples-operator-6748489656-nbbnq" (UID: "dc6432ef-0508-4d51-aba7-9331a7199df3") : secret "samples-operator-tls" not found Jul 24 15:48:01.263449 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.263352 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls podName:3685788b-ec3f-4d48-9bf8-132acf4b90bf nodeName:}" failed. No retries permitted until 2026-07-24 15:48:05.263333659 +0000 UTC m=+53.784942182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls") pod "image-registry-cd86f67cd-56fck" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf") : secret "image-registry-tls" not found Jul 24 15:48:01.667619 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.667580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.667780 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:01.667636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:01.667845 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.667823 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:48:01.667923 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.667911 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:02.667891172 +0000 UTC m=+51.189499690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : configmap "kube-rbac-proxy" not found Jul 24 15:48:01.667990 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.667927 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:48:01.668026 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:01.668000 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:02.667986735 +0000 UTC m=+51.189595275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : secret "insights-runtime-extractor-tls" not found Jul 24 15:48:02.210761 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.210674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" event={"ID":"a4980202-43a2-43ea-a4db-90e02a50ff32","Type":"ContainerStarted","Data":"608aab468f57e4253fbb5573da51063b6ea304c24328b6422404dc292cbf041c"} Jul 24 15:48:02.212452 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.212427 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/0.log" Jul 24 15:48:02.212567 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.212466 2580 generic.go:356] "Generic (PLEG): container finished" podID="f014c88e-fe9e-4ed5-9579-9c5f721bc6ff" containerID="bb253ba61b608df3c07fc2ea18f71dd524d83d5fed6102e5b80560f3132e511b" exitCode=255 Jul 24 15:48:02.212567 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.212547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" event={"ID":"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff","Type":"ContainerDied","Data":"bb253ba61b608df3c07fc2ea18f71dd524d83d5fed6102e5b80560f3132e511b"} Jul 24 15:48:02.212737 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.212716 2580 scope.go:117] "RemoveContainer" containerID="bb253ba61b608df3c07fc2ea18f71dd524d83d5fed6102e5b80560f3132e511b" Jul 24 15:48:02.214910 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.214887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" event={"ID":"aa78a585-835c-49d6-85b5-caeae5da578c","Type":"ContainerStarted","Data":"2e428b36ff62d1e9110b9ef43b192f0526fc76e7cfb8415c6e1d1ba532661fed"} Jul 24 15:48:02.216533 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.216512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" event={"ID":"563706c0-a505-4373-b987-923b16eb34da","Type":"ContainerStarted","Data":"1002b2f1c3b448a06aa98e678aa16c85bf14a812ae1591ac843a90bd6554e405"} Jul 24 15:48:02.225170 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.225134 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" podStartSLOduration=1.1591611689999999 podStartE2EDuration="5.225121949s" totalImagesPullingTime="4.06596078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:02.223845055 +0000 UTC m=+50.745453594" watchObservedRunningTime="2026-07-24 15:48:02.225121949 +0000 UTC m=+50.746730483" Jul 24 15:48:02.256504 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.256454 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-flr6s" podStartSLOduration=1.052040353 podStartE2EDuration="5.256438157s" totalImagesPullingTime="4.204397804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:02.237561615 +0000 UTC m=+50.759170156" watchObservedRunningTime="2026-07-24 15:48:02.256438157 +0000 UTC m=+50.778046676" Jul 24 15:48:02.271952 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.271910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:48:02.272607 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.272558 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" podStartSLOduration=1.18938595 podStartE2EDuration="5.272546784s" totalImagesPullingTime="4.083160834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:02.272432996 +0000 UTC m=+50.794041538" watchObservedRunningTime="2026-07-24 15:48:02.272546784 +0000 UTC m=+50.794155322" Jul 24 15:48:02.272725 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:02.272663 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:48:02.272725 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:02.272721 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls podName:2f27ac16-5144-4a65-9a33-791650b37e8c nodeName:}" failed. No retries permitted until 2026-07-24 15:48:18.272705076 +0000 UTC m=+66.794313600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls") pod "dns-default-fw7z6" (UID: "2f27ac16-5144-4a65-9a33-791650b37e8c") : secret "dns-default-metrics-tls" not found Jul 24 15:48:02.676432 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.676390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:02.676622 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:02.676451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:02.676622 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:02.676523 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:48:02.676622 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:02.676546 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:48:02.676622 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:02.676605 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:04.676584902 +0000 UTC m=+53.198193424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : configmap "kube-rbac-proxy" not found Jul 24 15:48:02.676622 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:02.676624 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:04.676614262 +0000 UTC m=+53.198222781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : secret "insights-runtime-extractor-tls" not found Jul 24 15:48:03.074497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.074468 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xmbr7_646c947d-c586-4522-9712-0204b10c4992/dns-node-resolver/0.log" Jul 24 15:48:03.222076 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.222049 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 15:48:03.222513 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.222497 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/0.log" Jul 24 15:48:03.222558 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.222539 2580 generic.go:356] "Generic (PLEG): container finished" podID="f014c88e-fe9e-4ed5-9579-9c5f721bc6ff" containerID="4049799de2f8d06a1e57b5fefac77c1f4350b6e8d40c2c002c439453dc62989b" exitCode=255 Jul 24 15:48:03.222688 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.222666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" event={"ID":"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff","Type":"ContainerDied","Data":"4049799de2f8d06a1e57b5fefac77c1f4350b6e8d40c2c002c439453dc62989b"} Jul 24 15:48:03.222759 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.222747 2580 scope.go:117] "RemoveContainer" containerID="bb253ba61b608df3c07fc2ea18f71dd524d83d5fed6102e5b80560f3132e511b" Jul 24 15:48:03.223229 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.223205 2580 scope.go:117] "RemoveContainer" containerID="4049799de2f8d06a1e57b5fefac77c1f4350b6e8d40c2c002c439453dc62989b" Jul 24 15:48:03.223419 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:03.223394 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vzvhl_openshift-console-operator(f014c88e-fe9e-4ed5-9579-9c5f721bc6ff)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" podUID="f014c88e-fe9e-4ed5-9579-9c5f721bc6ff" Jul 24 15:48:03.674242 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:03.674212 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gfbz8_3b687088-c49c-4cdb-a4bd-d0099456e00d/node-ca/0.log" Jul 24 15:48:04.226432 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:04.226405 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 15:48:04.226851 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:04.226682 2580 scope.go:117] "RemoveContainer" containerID="4049799de2f8d06a1e57b5fefac77c1f4350b6e8d40c2c002c439453dc62989b" Jul 24 15:48:04.226851 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:04.226838 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vzvhl_openshift-console-operator(f014c88e-fe9e-4ed5-9579-9c5f721bc6ff)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" podUID="f014c88e-fe9e-4ed5-9579-9c5f721bc6ff" Jul 24 15:48:04.679873 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:04.679843 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srv29_8e0c010d-8dda-4460-89d2-4fcf81149a91/serve-healthcheck-canary/0.log" Jul 24 15:48:04.692652 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:04.692621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:04.692774 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:04.692684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:04.692774 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:04.692750 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:48:04.692854 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:04.692796 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:48:04.692854 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:04.692815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:08.692799643 +0000 UTC m=+57.214408165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : secret "insights-runtime-extractor-tls" not found Jul 24 15:48:04.692931 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:04.692859 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:08.692847438 +0000 UTC m=+57.214455955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : configmap "kube-rbac-proxy" not found Jul 24 15:48:05.297345 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:05.297313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:48:05.297699 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:05.297358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:48:05.297699 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:05.297447 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:48:05.297699 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:05.297452 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:48:05.297699 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:05.297468 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd86f67cd-56fck: secret "image-registry-tls" not found Jul 24 15:48:05.297699 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:05.297505 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls podName:dc6432ef-0508-4d51-aba7-9331a7199df3 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:13.297491645 +0000 UTC m=+61.819100166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls") pod "cluster-samples-operator-6748489656-nbbnq" (UID: "dc6432ef-0508-4d51-aba7-9331a7199df3") : secret "samples-operator-tls" not found Jul 24 15:48:05.297699 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:05.297522 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls podName:3685788b-ec3f-4d48-9bf8-132acf4b90bf nodeName:}" failed. No retries permitted until 2026-07-24 15:48:13.297510062 +0000 UTC m=+61.819118580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls") pod "image-registry-cd86f67cd-56fck" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf") : secret "image-registry-tls" not found Jul 24 15:48:07.666546 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:07.666518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:48:07.666995 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:07.666569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:48:07.666995 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:07.666855 2580 scope.go:117] "RemoveContainer" containerID="4049799de2f8d06a1e57b5fefac77c1f4350b6e8d40c2c002c439453dc62989b" Jul 24 15:48:07.667067 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:07.667016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vzvhl_openshift-console-operator(f014c88e-fe9e-4ed5-9579-9c5f721bc6ff)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" podUID="f014c88e-fe9e-4ed5-9579-9c5f721bc6ff" Jul 24 15:48:08.722672 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:08.722626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:08.723095 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:08.722701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:08.723095 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:08.722748 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:48:08.723095 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:08.722771 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:48:08.723095 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:08.722817 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:16.722799 +0000 UTC m=+65.244407536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : configmap "kube-rbac-proxy" not found Jul 24 15:48:08.723095 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:08.722832 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:16.722823992 +0000 UTC m=+65.244432510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : secret "insights-runtime-extractor-tls" not found Jul 24 15:48:12.163358 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:12.163323 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb6dp" Jul 24 15:48:13.360228 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:13.360187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:48:13.360696 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:13.360299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") pod \"image-registry-cd86f67cd-56fck\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:48:13.360696 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:13.360390 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:48:13.360696 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:13.360401 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cd86f67cd-56fck: secret "image-registry-tls" not found Jul 24 15:48:13.360696 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:13.360449 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls podName:3685788b-ec3f-4d48-9bf8-132acf4b90bf nodeName:}" failed. No retries permitted until 2026-07-24 15:48:29.360434596 +0000 UTC m=+77.882043114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls") pod "image-registry-cd86f67cd-56fck" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf") : secret "image-registry-tls" not found Jul 24 15:48:13.362820 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:13.362795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc6432ef-0508-4d51-aba7-9331a7199df3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nbbnq\" (UID: \"dc6432ef-0508-4d51-aba7-9331a7199df3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:48:13.380460 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:13.380434 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l9bfq\"" Jul 24 15:48:13.388884 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:13.388860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" Jul 24 15:48:13.520902 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:13.520871 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq"] Jul 24 15:48:14.252278 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:14.252230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" event={"ID":"dc6432ef-0508-4d51-aba7-9331a7199df3","Type":"ContainerStarted","Data":"3751e76b6c32f23d0e7efda7f3faffce6357100e64c60f706eac10f964b8e185"} Jul 24 15:48:15.257484 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:15.257450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" event={"ID":"dc6432ef-0508-4d51-aba7-9331a7199df3","Type":"ContainerStarted","Data":"e421512fb27578bf092260316be8c0b8769b7fe6ff4bb52f479748610a42b67f"} Jul 24 15:48:15.257860 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:15.257520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" event={"ID":"dc6432ef-0508-4d51-aba7-9331a7199df3","Type":"ContainerStarted","Data":"ea08eaaacacf0470ba5ca33935957d220b1b0a7a5a61c16110aa0158e727e89f"} Jul 24 15:48:15.272407 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:15.272359 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nbbnq" podStartSLOduration=16.771610733 podStartE2EDuration="18.272342121s" totalImagesPullingTime="1.500731388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:15.271672818 +0000 UTC m=+63.793281357" watchObservedRunningTime="2026-07-24 15:48:15.272342121 +0000 UTC m=+63.793950660" Jul 24 15:48:16.786617 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:16.786581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:16.787148 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:16.786642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:16.787148 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:16.786743 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:48:16.787148 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:16.786814 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls podName:99e0f12c-caff-4e63-a132-ed8cea3448b9 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:32.786795646 +0000 UTC m=+81.308404167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvvwq" (UID: "99e0f12c-caff-4e63-a132-ed8cea3448b9") : secret "insights-runtime-extractor-tls" not found Jul 24 15:48:16.787339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:16.787291 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e0f12c-caff-4e63-a132-ed8cea3448b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:17.693163 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.693119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:48:17.695339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.695321 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:48:17.704014 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:17.703994 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 15:48:17.704084 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:17.704066 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs podName:1837b9cd-372b-4e0e-ab2a-ff6013c26aea nodeName:}" failed. No retries permitted until 2026-07-24 15:49:21.704049248 +0000 UTC m=+130.225657765 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs") pod "network-metrics-daemon-ngh44" (UID: "1837b9cd-372b-4e0e-ab2a-ff6013c26aea") : secret "metrics-daemon-secret" not found Jul 24 15:48:17.794317 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.794284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:48:17.794672 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.794329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:48:17.796684 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.796662 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:48:17.796759 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.796684 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:48:17.806799 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.806772 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:48:17.807588 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.807572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3c5e7b6-0692-4501-883c-29b33100cbd9-original-pull-secret\") pod \"global-pull-secret-syncer-txr7v\" (UID: \"b3c5e7b6-0692-4501-883c-29b33100cbd9\") " pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:48:17.818724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.818699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-256v8\" (UniqueName: \"kubernetes.io/projected/4c3aa307-3e72-44fb-bd78-b6ca76736bd2-kube-api-access-256v8\") pod \"network-check-target-6lqpv\" (UID: \"4c3aa307-3e72-44fb-bd78-b6ca76736bd2\") " pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:48:17.916114 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:17.916083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txr7v" Jul 24 15:48:18.018182 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.018155 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w7pwm\"" Jul 24 15:48:18.027018 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.026993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:48:18.037904 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.037872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txr7v"] Jul 24 15:48:18.041278 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:18.041250 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c5e7b6_0692_4501_883c_29b33100cbd9.slice/crio-7331701d606a6d76240622e232b61cb796c58aa8207696cac64f5b7c906b04f8 WatchSource:0}: Error finding container 7331701d606a6d76240622e232b61cb796c58aa8207696cac64f5b7c906b04f8: Status 404 returned error can't find the container with id 7331701d606a6d76240622e232b61cb796c58aa8207696cac64f5b7c906b04f8 Jul 24 15:48:18.152019 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.151986 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6lqpv"] Jul 24 15:48:18.154998 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:18.154970 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3aa307_3e72_44fb_bd78_b6ca76736bd2.slice/crio-08deac370aada8c9c8985bec955dc35602a8c92b05ebd420ad0c6cfa00a0fcbc WatchSource:0}: Error finding container 08deac370aada8c9c8985bec955dc35602a8c92b05ebd420ad0c6cfa00a0fcbc: Status 404 returned error can't find the container with id 08deac370aada8c9c8985bec955dc35602a8c92b05ebd420ad0c6cfa00a0fcbc Jul 24 15:48:18.272357 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.272314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txr7v" event={"ID":"b3c5e7b6-0692-4501-883c-29b33100cbd9","Type":"ContainerStarted","Data":"7331701d606a6d76240622e232b61cb796c58aa8207696cac64f5b7c906b04f8"} Jul 24 15:48:18.273482 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.273447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6lqpv" event={"ID":"4c3aa307-3e72-44fb-bd78-b6ca76736bd2","Type":"ContainerStarted","Data":"08deac370aada8c9c8985bec955dc35602a8c92b05ebd420ad0c6cfa00a0fcbc"} Jul 24 15:48:18.298109 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.298085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:48:18.300549 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.300528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f27ac16-5144-4a65-9a33-791650b37e8c-metrics-tls\") pod \"dns-default-fw7z6\" (UID: \"2f27ac16-5144-4a65-9a33-791650b37e8c\") " pod="openshift-dns/dns-default-fw7z6" Jul 24 15:48:18.527435 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.527357 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59fhg\"" Jul 24 15:48:18.535885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.535861 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fw7z6" Jul 24 15:48:18.663983 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:18.663951 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fw7z6"] Jul 24 15:48:18.666714 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:18.666685 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f27ac16_5144_4a65_9a33_791650b37e8c.slice/crio-0325aa0afb08ea83026f6288540266603695928a545c33980f6f8b6db3f3a9f1 WatchSource:0}: Error finding container 0325aa0afb08ea83026f6288540266603695928a545c33980f6f8b6db3f3a9f1: Status 404 returned error can't find the container with id 0325aa0afb08ea83026f6288540266603695928a545c33980f6f8b6db3f3a9f1 Jul 24 15:48:19.279976 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:19.279917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fw7z6" event={"ID":"2f27ac16-5144-4a65-9a33-791650b37e8c","Type":"ContainerStarted","Data":"0325aa0afb08ea83026f6288540266603695928a545c33980f6f8b6db3f3a9f1"} Jul 24 15:48:20.007595 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:20.007568 2580 scope.go:117] "RemoveContainer" containerID="4049799de2f8d06a1e57b5fefac77c1f4350b6e8d40c2c002c439453dc62989b" Jul 24 15:48:20.287652 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:20.287574 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 15:48:20.288030 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:20.287780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" event={"ID":"f014c88e-fe9e-4ed5-9579-9c5f721bc6ff","Type":"ContainerStarted","Data":"18346eec59fe6682cb6efe3c462e3609f70a6a8b720a8e5d36e59a9e1125941a"} Jul 24 15:48:20.289030 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:20.289009 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:48:20.304854 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:20.304778 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" podStartSLOduration=19.201955973 podStartE2EDuration="23.304759502s" totalImagesPullingTime="4.102803529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:20.303487273 +0000 UTC m=+68.825095835" watchObservedRunningTime="2026-07-24 15:48:20.304759502 +0000 UTC m=+68.826368042" Jul 24 15:48:21.136618 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:21.136585 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-vzvhl" Jul 24 15:48:24.303458 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.303417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fw7z6" event={"ID":"2f27ac16-5144-4a65-9a33-791650b37e8c","Type":"ContainerStarted","Data":"c26066d6a3b67aae03105f9b65885a79f5862f1bc5cdc7c2dde723217f7d3e7b"} Jul 24 15:48:24.303964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.303475 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fw7z6" event={"ID":"2f27ac16-5144-4a65-9a33-791650b37e8c","Type":"ContainerStarted","Data":"7b82349da92c82484ab5c6018c9a67753a5e52029782aa204d3e329de1e47586"} Jul 24 15:48:24.303964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.303499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fw7z6" Jul 24 15:48:24.304925 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.304905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txr7v" event={"ID":"b3c5e7b6-0692-4501-883c-29b33100cbd9","Type":"ContainerStarted","Data":"c47d7decf11697fc3cc74f4b0d8766e582c0f6649878145f423d5ca7f4eaee49"} Jul 24 15:48:24.306445 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.306419 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6lqpv" event={"ID":"4c3aa307-3e72-44fb-bd78-b6ca76736bd2","Type":"ContainerStarted","Data":"4744dbc224bc5f9442f0e5161ade4af9bf1da20c21126fcc0f06dea8b2ba5050"} Jul 24 15:48:24.306568 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.306547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:48:24.319849 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.319807 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fw7z6" podStartSLOduration=33.706384351 podStartE2EDuration="38.319790253s" totalImagesPullingTime="4.613405902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:24.318328362 +0000 UTC m=+72.839936893" watchObservedRunningTime="2026-07-24 15:48:24.319790253 +0000 UTC m=+72.841398795" Jul 24 15:48:24.331897 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.331854 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-txr7v" podStartSLOduration=67.075115046 podStartE2EDuration="1m12.331840684s" totalImagesPullingTime="5.256725638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:24.331007905 +0000 UTC m=+72.852616445" watchObservedRunningTime="2026-07-24 15:48:24.331840684 +0000 UTC m=+72.853449224" Jul 24 15:48:24.344360 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:24.344320 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6lqpv" podStartSLOduration=67.206413952 podStartE2EDuration="1m12.3443036s" totalImagesPullingTime="5.137889648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:24.343586584 +0000 UTC m=+72.865195114" watchObservedRunningTime="2026-07-24 15:48:24.3443036 +0000 UTC m=+72.865912134" Jul 24 15:48:25.115547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.115508 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr"] Jul 24 15:48:25.117445 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.117429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.120801 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.120775 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 15:48:25.121065 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.121052 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 15:48:25.121685 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.121663 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 15:48:25.121685 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.121680 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 15:48:25.143003 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.142980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr"] Jul 24 15:48:25.163183 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.163155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-klusterlet-config\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.163361 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.163310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256vm\" (UniqueName: \"kubernetes.io/projected/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-kube-api-access-256vm\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.163418 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.163382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-tmp\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.221434 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.221404 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cd86f67cd-56fck"] Jul 24 15:48:25.221636 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:25.221615 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-cd86f67cd-56fck" podUID="3685788b-ec3f-4d48-9bf8-132acf4b90bf" Jul 24 15:48:25.255957 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.255909 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cf7574fb5-697wp"] Jul 24 15:48:25.257882 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.257860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.263863 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.263839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-tmp\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.264037 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.263885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-klusterlet-config\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.264037 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.264002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256vm\" (UniqueName: \"kubernetes.io/projected/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-kube-api-access-256vm\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.264478 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.264266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-tmp\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.267263 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.267238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-klusterlet-config\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.270778 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.270739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf7574fb5-697wp"] Jul 24 15:48:25.279094 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.279074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-256vm\" (UniqueName: \"kubernetes.io/projected/dfd0f656-c6a5-477b-b75e-6a0c549b1f3e-kube-api-access-256vm\") pod \"klusterlet-addon-workmgr-655c98c648-wnjdr\" (UID: \"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.309551 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.309524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:48:25.315125 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.315107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:48:25.365159 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-installation-pull-secrets\") pod \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " Jul 24 15:48:25.365159 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365162 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vs94\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-kube-api-access-9vs94\") pod \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " Jul 24 15:48:25.365389 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365192 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-certificates\") pod \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " Jul 24 15:48:25.365389 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365219 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-bound-sa-token\") pod \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " Jul 24 15:48:25.365389 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365235 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-trusted-ca\") pod \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " Jul 24 15:48:25.365389 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365280 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-image-registry-private-configuration\") pod \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " Jul 24 15:48:25.365389 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365310 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3685788b-ec3f-4d48-9bf8-132acf4b90bf-ca-trust-extracted\") pod \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\" (UID: \"3685788b-ec3f-4d48-9bf8-132acf4b90bf\") " Jul 24 15:48:25.365665 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-bound-sa-token\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.365665 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cf83508-bfa7-467a-b016-f12406c82769-image-registry-private-configuration\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.365665 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-registry-tls\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.365665 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cf83508-bfa7-467a-b016-f12406c82769-registry-certificates\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.365665 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cf83508-bfa7-467a-b016-f12406c82769-trusted-ca\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.365665 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cf83508-bfa7-467a-b016-f12406c82769-installation-pull-secrets\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.366006 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cf83508-bfa7-467a-b016-f12406c82769-ca-trust-extracted\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.366006 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6t5\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-kube-api-access-bj6t5\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.366006 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365733 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3685788b-ec3f-4d48-9bf8-132acf4b90bf" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:25.366006 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365732 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3685788b-ec3f-4d48-9bf8-132acf4b90bf" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:25.366006 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.365975 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3685788b-ec3f-4d48-9bf8-132acf4b90bf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3685788b-ec3f-4d48-9bf8-132acf4b90bf" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:25.366424 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.366131 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-trusted-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:25.366424 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.366153 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-certificates\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:25.367911 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.367884 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3685788b-ec3f-4d48-9bf8-132acf4b90bf" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:25.367911 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.367884 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3685788b-ec3f-4d48-9bf8-132acf4b90bf" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:25.368086 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.368001 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3685788b-ec3f-4d48-9bf8-132acf4b90bf" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:25.368086 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.368008 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-kube-api-access-9vs94" (OuterVolumeSpecName: "kube-api-access-9vs94") pod "3685788b-ec3f-4d48-9bf8-132acf4b90bf" (UID: "3685788b-ec3f-4d48-9bf8-132acf4b90bf"). InnerVolumeSpecName "kube-api-access-9vs94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:25.426613 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.426587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:25.466765 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cf83508-bfa7-467a-b016-f12406c82769-installation-pull-secrets\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.466973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cf83508-bfa7-467a-b016-f12406c82769-ca-trust-extracted\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.466973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6t5\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-kube-api-access-bj6t5\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.466973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-bound-sa-token\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.466973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cf83508-bfa7-467a-b016-f12406c82769-image-registry-private-configuration\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.466973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-registry-tls\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.466973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cf83508-bfa7-467a-b016-f12406c82769-registry-certificates\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.466973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.466967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cf83508-bfa7-467a-b016-f12406c82769-trusted-ca\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.467370 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.467006 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-installation-pull-secrets\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:25.467370 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.467025 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vs94\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-kube-api-access-9vs94\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:25.467370 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.467041 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-bound-sa-token\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:25.467370 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.467057 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3685788b-ec3f-4d48-9bf8-132acf4b90bf-image-registry-private-configuration\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:25.467370 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.467070 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3685788b-ec3f-4d48-9bf8-132acf4b90bf-ca-trust-extracted\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:25.467370 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.467202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cf83508-bfa7-467a-b016-f12406c82769-ca-trust-extracted\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.468170 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.468133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cf83508-bfa7-467a-b016-f12406c82769-registry-certificates\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.468295 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.468134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cf83508-bfa7-467a-b016-f12406c82769-trusted-ca\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.469428 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.469400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cf83508-bfa7-467a-b016-f12406c82769-installation-pull-secrets\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.469553 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.469536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-registry-tls\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.469650 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.469635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cf83508-bfa7-467a-b016-f12406c82769-image-registry-private-configuration\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.477978 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.477745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-bound-sa-token\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.478346 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.478320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6t5\" (UniqueName: \"kubernetes.io/projected/1cf83508-bfa7-467a-b016-f12406c82769-kube-api-access-bj6t5\") pod \"image-registry-7cf7574fb5-697wp\" (UID: \"1cf83508-bfa7-467a-b016-f12406c82769\") " pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.554985 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.554930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr"] Jul 24 15:48:25.557653 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:25.557624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd0f656_c6a5_477b_b75e_6a0c549b1f3e.slice/crio-6b3a6348f2ca6bd7e8ba768f59f365d253cb969e682c91f25e51454557de7328 WatchSource:0}: Error finding container 6b3a6348f2ca6bd7e8ba768f59f365d253cb969e682c91f25e51454557de7328: Status 404 returned error can't find the container with id 6b3a6348f2ca6bd7e8ba768f59f365d253cb969e682c91f25e51454557de7328 Jul 24 15:48:25.569417 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.569392 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-phckq\"" Jul 24 15:48:25.578835 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.578818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:25.709574 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:25.709544 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf7574fb5-697wp"] Jul 24 15:48:25.712467 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:25.712445 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf83508_bfa7_467a_b016_f12406c82769.slice/crio-d7cdeefe1200492822d281dd136c414b9860ed58bbdcafe273aaa8d7f8b360f3 WatchSource:0}: Error finding container d7cdeefe1200492822d281dd136c414b9860ed58bbdcafe273aaa8d7f8b360f3: Status 404 returned error can't find the container with id d7cdeefe1200492822d281dd136c414b9860ed58bbdcafe273aaa8d7f8b360f3 Jul 24 15:48:26.315420 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.315316 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" event={"ID":"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e","Type":"ContainerStarted","Data":"6b3a6348f2ca6bd7e8ba768f59f365d253cb969e682c91f25e51454557de7328"} Jul 24 15:48:26.317443 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.317404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" event={"ID":"1cf83508-bfa7-467a-b016-f12406c82769","Type":"ContainerStarted","Data":"dbb165603a03f64c479cea2507305874278a61512e055f9b917ca898fdc13bb7"} Jul 24 15:48:26.317564 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.317465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" event={"ID":"1cf83508-bfa7-467a-b016-f12406c82769","Type":"ContainerStarted","Data":"d7cdeefe1200492822d281dd136c414b9860ed58bbdcafe273aaa8d7f8b360f3"} Jul 24 15:48:26.317564 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.317533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:26.317564 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.317562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd86f67cd-56fck" Jul 24 15:48:26.337140 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.337087 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" podStartSLOduration=1.337069919 podStartE2EDuration="1.337069919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:26.335475255 +0000 UTC m=+74.857083807" watchObservedRunningTime="2026-07-24 15:48:26.337069919 +0000 UTC m=+74.858678458" Jul 24 15:48:26.360284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.360248 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cd86f67cd-56fck"] Jul 24 15:48:26.363083 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.363054 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cd86f67cd-56fck"] Jul 24 15:48:26.475712 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:26.475679 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3685788b-ec3f-4d48-9bf8-132acf4b90bf-registry-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:48:28.008779 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:28.008746 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3685788b-ec3f-4d48-9bf8-132acf4b90bf" path="/var/lib/kubelet/pods/3685788b-ec3f-4d48-9bf8-132acf4b90bf/volumes" Jul 24 15:48:29.330295 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:29.330255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" event={"ID":"dfd0f656-c6a5-477b-b75e-6a0c549b1f3e","Type":"ContainerStarted","Data":"86995ef8eb06229704f33e043e6420ea944e4c0f5a19ac9b21000d04b6cca41a"} Jul 24 15:48:29.330711 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:29.330480 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:29.332147 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:29.332125 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" Jul 24 15:48:29.347823 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:29.347777 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655c98c648-wnjdr" podStartSLOduration=0.750509744 podStartE2EDuration="4.347760858s" totalImagesPullingTime="3.597251114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:29.347019273 +0000 UTC m=+77.868627827" watchObservedRunningTime="2026-07-24 15:48:29.347760858 +0000 UTC m=+77.869369398" Jul 24 15:48:32.517964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.517908 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp"] Jul 24 15:48:32.522866 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.522842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:32.524662 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.524638 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 15:48:32.524781 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.524698 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bvmfl\"" Jul 24 15:48:32.530643 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.530623 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp"] Jul 24 15:48:32.627642 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.627606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/288a192f-c4e8-4f57-b724-2787cee4b953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-jc2hp\" (UID: \"288a192f-c4e8-4f57-b724-2787cee4b953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:32.728603 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.728564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/288a192f-c4e8-4f57-b724-2787cee4b953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-jc2hp\" (UID: \"288a192f-c4e8-4f57-b724-2787cee4b953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:32.728779 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:32.728727 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 24 15:48:32.728827 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:32.728818 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288a192f-c4e8-4f57-b724-2787cee4b953-tls-certificates podName:288a192f-c4e8-4f57-b724-2787cee4b953 nodeName:}" failed. No retries permitted until 2026-07-24 15:48:33.228800675 +0000 UTC m=+81.750409193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/288a192f-c4e8-4f57-b724-2787cee4b953-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" (UID: "288a192f-c4e8-4f57-b724-2787cee4b953") : secret "prometheus-operator-admission-webhook-tls" not found Jul 24 15:48:32.829763 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.829669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:32.832164 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:32.832143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e0f12c-caff-4e63-a132-ed8cea3448b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvvwq\" (UID: \"99e0f12c-caff-4e63-a132-ed8cea3448b9\") " pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:33.012811 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.012779 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qb9zg\"" Jul 24 15:48:33.021676 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.021553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvvwq" Jul 24 15:48:33.175954 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.175832 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvvwq"] Jul 24 15:48:33.179663 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:33.179624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e0f12c_caff_4e63_a132_ed8cea3448b9.slice/crio-36b6db8df6bfecb70ac3ce1e1d7e10325fbf5cba1060174583b1ed92530362ac WatchSource:0}: Error finding container 36b6db8df6bfecb70ac3ce1e1d7e10325fbf5cba1060174583b1ed92530362ac: Status 404 returned error can't find the container with id 36b6db8df6bfecb70ac3ce1e1d7e10325fbf5cba1060174583b1ed92530362ac Jul 24 15:48:33.233404 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.233373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/288a192f-c4e8-4f57-b724-2787cee4b953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-jc2hp\" (UID: \"288a192f-c4e8-4f57-b724-2787cee4b953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:33.236036 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.236014 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/288a192f-c4e8-4f57-b724-2787cee4b953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-jc2hp\" (UID: \"288a192f-c4e8-4f57-b724-2787cee4b953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:33.348219 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.348130 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvvwq" event={"ID":"99e0f12c-caff-4e63-a132-ed8cea3448b9","Type":"ContainerStarted","Data":"36b6db8df6bfecb70ac3ce1e1d7e10325fbf5cba1060174583b1ed92530362ac"} Jul 24 15:48:33.431754 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.431709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:33.616858 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:33.616774 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp"] Jul 24 15:48:33.620411 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:33.620381 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a192f_c4e8_4f57_b724_2787cee4b953.slice/crio-fe1744c0ca57e2d092652c9ce970f417933196a865aba7386b642e3c173bb7ba WatchSource:0}: Error finding container fe1744c0ca57e2d092652c9ce970f417933196a865aba7386b642e3c173bb7ba: Status 404 returned error can't find the container with id fe1744c0ca57e2d092652c9ce970f417933196a865aba7386b642e3c173bb7ba Jul 24 15:48:34.312486 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:34.311929 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fw7z6" Jul 24 15:48:34.354901 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:34.354862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" event={"ID":"288a192f-c4e8-4f57-b724-2787cee4b953","Type":"ContainerStarted","Data":"fe1744c0ca57e2d092652c9ce970f417933196a865aba7386b642e3c173bb7ba"} Jul 24 15:48:35.360550 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:35.360503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" event={"ID":"288a192f-c4e8-4f57-b724-2787cee4b953","Type":"ContainerStarted","Data":"6ee5d662ee82bf0a9e38ae31c96df7662c0d0e2d8b10fe9a60709ac78a6f3960"} Jul 24 15:48:35.361015 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:35.360713 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:35.366297 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:35.366274 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" Jul 24 15:48:35.374388 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:35.374302 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-jc2hp" podStartSLOduration=2.187140215 podStartE2EDuration="3.374285524s" totalImagesPullingTime="1.187145309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:35.373870637 +0000 UTC m=+83.895479178" watchObservedRunningTime="2026-07-24 15:48:35.374285524 +0000 UTC m=+83.895894065" Jul 24 15:48:38.380041 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:38.379953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvvwq" event={"ID":"99e0f12c-caff-4e63-a132-ed8cea3448b9","Type":"ContainerStarted","Data":"ed9c85b0e9769dfb3b28a10829ad7001d2f9cf0b2fe72f2487e158651d39f550"} Jul 24 15:48:42.397798 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:42.397746 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvvwq" event={"ID":"99e0f12c-caff-4e63-a132-ed8cea3448b9","Type":"ContainerStarted","Data":"e183a6cdd88542461e0b3fcbb9bb6dcfda64b0a70935a80198ac74a9e14c1fc9"} Jul 24 15:48:46.414575 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:46.414541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvvwq" event={"ID":"99e0f12c-caff-4e63-a132-ed8cea3448b9","Type":"ContainerStarted","Data":"d5b4e9be73673104369b41b146fa4457e36cf8e77b7658a0c75e55f1bb842c00"} Jul 24 15:48:46.432886 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:46.432826 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mvvwq" podStartSLOduration=33.920508789 podStartE2EDuration="46.432806403s" totalImagesPullingTime="12.512297614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:48:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:46.430852593 +0000 UTC m=+94.952461156" watchObservedRunningTime="2026-07-24 15:48:46.432806403 +0000 UTC m=+94.954414944" Jul 24 15:48:47.325618 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:47.325591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cf7574fb5-697wp" Jul 24 15:48:55.311846 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:55.311819 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6lqpv" Jul 24 15:48:56.956213 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.956173 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sczkq"] Jul 24 15:48:56.962815 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.962787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:56.965230 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.965201 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 15:48:56.965390 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.965202 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 15:48:56.965390 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.965209 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twmxq\"" Jul 24 15:48:56.965390 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.965254 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 15:48:56.965730 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.965708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 15:48:56.965953 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.965736 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 15:48:56.965953 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:56.965754 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 15:48:57.018627 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-wtmp\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018769 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018769 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-textfile\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018769 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85745cba-b8f4-4928-a9c6-65cc9ae288ac-metrics-client-ca\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018769 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-tls\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-sys\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-root\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.018964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.018865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhcx\" (UniqueName: \"kubernetes.io/projected/85745cba-b8f4-4928-a9c6-65cc9ae288ac-kube-api-access-pbhcx\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119401 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-wtmp\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-textfile\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85745cba-b8f4-4928-a9c6-65cc9ae288ac-metrics-client-ca\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-tls\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-sys\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-root\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119571 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-wtmp\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhcx\" (UniqueName: \"kubernetes.io/projected/85745cba-b8f4-4928-a9c6-65cc9ae288ac-kube-api-access-pbhcx\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-sys\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85745cba-b8f4-4928-a9c6-65cc9ae288ac-root\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.119885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.119853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-textfile\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.120195 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.120159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.120304 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.120255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85745cba-b8f4-4928-a9c6-65cc9ae288ac-metrics-client-ca\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.122248 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.122229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-tls\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.122350 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.122269 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85745cba-b8f4-4928-a9c6-65cc9ae288ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.126949 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.126907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhcx\" (UniqueName: \"kubernetes.io/projected/85745cba-b8f4-4928-a9c6-65cc9ae288ac-kube-api-access-pbhcx\") pod \"node-exporter-sczkq\" (UID: \"85745cba-b8f4-4928-a9c6-65cc9ae288ac\") " pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.272479 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.272442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sczkq" Jul 24 15:48:57.312721 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:57.312689 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85745cba_b8f4_4928_a9c6_65cc9ae288ac.slice/crio-22018db16ed79b66f4b6892183fe475841f61c8f6ece2f698d0fd70291e0ab19 WatchSource:0}: Error finding container 22018db16ed79b66f4b6892183fe475841f61c8f6ece2f698d0fd70291e0ab19: Status 404 returned error can't find the container with id 22018db16ed79b66f4b6892183fe475841f61c8f6ece2f698d0fd70291e0ab19 Jul 24 15:48:57.452090 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.452049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sczkq" event={"ID":"85745cba-b8f4-4928-a9c6-65cc9ae288ac","Type":"ContainerStarted","Data":"22018db16ed79b66f4b6892183fe475841f61c8f6ece2f698d0fd70291e0ab19"} Jul 24 15:48:57.996790 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:57.996761 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:58.023565 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.023533 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.025763 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.025732 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 15:48:58.025763 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.025773 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 15:48:58.026019 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.025920 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 15:48:58.026019 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.025979 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 15:48:58.026171 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026088 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 15:48:58.026229 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026195 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fslvz\"" Jul 24 15:48:58.026278 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026234 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 15:48:58.026445 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026389 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 15:48:58.026528 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026470 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 15:48:58.027053 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026832 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 15:48:58.027053 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027053 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.026970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027053 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-config-out\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997xj\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-kube-api-access-997xj\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-web-config\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.027403 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.027395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.029274 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.029248 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:58.128268 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128268 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128522 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128522 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:48:58.128502 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle podName:0090adb5-2358-499a-bb2e-9416787e5a1c nodeName:}" failed. No retries permitted until 2026-07-24 15:48:58.628479564 +0000 UTC m=+107.150088099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c") : configmap references non-existent config key: ca-bundle.crt Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-config-out\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997xj\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-kube-api-access-997xj\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-web-config\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.128786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.129281 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.128812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.129341 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.129304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.129992 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.129727 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.133010 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.132980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.133991 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.133380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.133991 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.133417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-config-out\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.133991 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.133586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.133991 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.133889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.133991 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.133923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.134359 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.134314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.134675 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.134657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.136225 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.136208 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-web-config\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.139499 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.139473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-997xj\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-kube-api-access-997xj\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.632841 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.632815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.633640 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.633618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.636805 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.636788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:58.812722 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.812691 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:58.816526 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:58.816456 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0090adb5_2358_499a_bb2e_9416787e5a1c.slice/crio-a21583ea74ca8fc0f7164353a8978aa7bb81de7b2801d1324fe24ba2fa49fe9f WatchSource:0}: Error finding container a21583ea74ca8fc0f7164353a8978aa7bb81de7b2801d1324fe24ba2fa49fe9f: Status 404 returned error can't find the container with id a21583ea74ca8fc0f7164353a8978aa7bb81de7b2801d1324fe24ba2fa49fe9f Jul 24 15:48:58.904147 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.904072 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7cb95767d6-jnzr9"] Jul 24 15:48:58.916753 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.916726 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.918786 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.918761 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 24 15:48:58.918913 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.918784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cb95767d6-jnzr9"] Jul 24 15:48:58.918913 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.918775 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 24 15:48:58.919041 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.918912 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7sc2ectjhsc79\"" Jul 24 15:48:58.919041 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.918963 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 24 15:48:58.919041 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.919032 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 24 15:48:58.919175 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.919042 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 24 15:48:58.919175 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.919054 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pfhv4\"" Jul 24 15:48:58.936180 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.936294 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.936294 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-tls\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.936363 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.936363 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.936363 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hjz\" (UniqueName: \"kubernetes.io/projected/301f70fa-048d-48e6-b818-bb9f3f2a6243-kube-api-access-57hjz\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.936508 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-grpc-tls\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:58.936508 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:58.936445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/301f70fa-048d-48e6-b818-bb9f3f2a6243-metrics-client-ca\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.037885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.037853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.037885 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.037887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.038415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.037913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hjz\" (UniqueName: \"kubernetes.io/projected/301f70fa-048d-48e6-b818-bb9f3f2a6243-kube-api-access-57hjz\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.038415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.037954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-grpc-tls\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.038415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.037989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/301f70fa-048d-48e6-b818-bb9f3f2a6243-metrics-client-ca\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.038415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.038027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.038415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.038063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.038415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.038211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-tls\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.038930 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.038877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/301f70fa-048d-48e6-b818-bb9f3f2a6243-metrics-client-ca\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.041034 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.041010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.041120 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.041070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.041192 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.041174 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.041242 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.041226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.041332 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.041315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-grpc-tls\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.041538 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.041525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/301f70fa-048d-48e6-b818-bb9f3f2a6243-secret-thanos-querier-tls\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.045684 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.045623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hjz\" (UniqueName: \"kubernetes.io/projected/301f70fa-048d-48e6-b818-bb9f3f2a6243-kube-api-access-57hjz\") pod \"thanos-querier-7cb95767d6-jnzr9\" (UID: \"301f70fa-048d-48e6-b818-bb9f3f2a6243\") " pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.227434 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.227339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:48:59.393670 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.393634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cb95767d6-jnzr9"] Jul 24 15:48:59.397232 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:48:59.397198 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301f70fa_048d_48e6_b818_bb9f3f2a6243.slice/crio-4af6337b9505cebfa14641f50076bffa706bbb1f0e3f06c615b0d76bcaa25b7b WatchSource:0}: Error finding container 4af6337b9505cebfa14641f50076bffa706bbb1f0e3f06c615b0d76bcaa25b7b: Status 404 returned error can't find the container with id 4af6337b9505cebfa14641f50076bffa706bbb1f0e3f06c615b0d76bcaa25b7b Jul 24 15:48:59.464121 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.464085 2580 generic.go:356] "Generic (PLEG): container finished" podID="85745cba-b8f4-4928-a9c6-65cc9ae288ac" containerID="f09a363b7f85a2847ab2e7c0fd650112843d28cf73ad7566e64bb1db999b1114" exitCode=0 Jul 24 15:48:59.464304 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.464171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sczkq" event={"ID":"85745cba-b8f4-4928-a9c6-65cc9ae288ac","Type":"ContainerDied","Data":"f09a363b7f85a2847ab2e7c0fd650112843d28cf73ad7566e64bb1db999b1114"} Jul 24 15:48:59.465440 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.465415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" event={"ID":"301f70fa-048d-48e6-b818-bb9f3f2a6243","Type":"ContainerStarted","Data":"4af6337b9505cebfa14641f50076bffa706bbb1f0e3f06c615b0d76bcaa25b7b"} Jul 24 15:48:59.466580 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:48:59.466553 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerStarted","Data":"a21583ea74ca8fc0f7164353a8978aa7bb81de7b2801d1324fe24ba2fa49fe9f"} Jul 24 15:49:00.472382 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:00.472353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sczkq" event={"ID":"85745cba-b8f4-4928-a9c6-65cc9ae288ac","Type":"ContainerStarted","Data":"94863b9cdf8ee1810a10a295bc702d7c5550dedf8ce1787c3cabe7adf472d39b"} Jul 24 15:49:00.472747 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:00.472415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sczkq" event={"ID":"85745cba-b8f4-4928-a9c6-65cc9ae288ac","Type":"ContainerStarted","Data":"3265d328154711056e4929625039c7bf14110bb153c51acc77496f01db72ecda"} Jul 24 15:49:00.492766 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:00.492722 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sczkq" podStartSLOduration=3.119877741 podStartE2EDuration="4.492705144s" totalImagesPullingTime="1.274129696s" totalInitContainerRuntime="98.697707ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:00.491248577 +0000 UTC m=+109.012857132" watchObservedRunningTime="2026-07-24 15:49:00.492705144 +0000 UTC m=+109.014313684" Jul 24 15:49:01.228946 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.228916 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5fc4d6979c-j5tfg"] Jul 24 15:49:01.238963 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.238495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.240228 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.240156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fc4d6979c-j5tfg"] Jul 24 15:49:01.241964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.241300 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ztrdk\"" Jul 24 15:49:01.241964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.241331 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 15:49:01.241964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.241349 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 15:49:01.241964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.241396 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 15:49:01.241964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.241337 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 15:49:01.241964 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.241617 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6c1ld31hqh15g\"" Jul 24 15:49:01.256363 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.256340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-client-ca-bundle\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.256478 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.256381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc2ec951-5b31-401a-865e-6dc40fbff02b-metrics-server-audit-profiles\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.256478 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.256420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrksx\" (UniqueName: \"kubernetes.io/projected/cc2ec951-5b31-401a-865e-6dc40fbff02b-kube-api-access-xrksx\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.256478 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.256461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-secret-metrics-server-client-certs\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.256478 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.256479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-secret-metrics-server-tls\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.256677 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.256501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc2ec951-5b31-401a-865e-6dc40fbff02b-audit-log\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.256677 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.256583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2ec951-5b31-401a-865e-6dc40fbff02b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.357860 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.357827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2ec951-5b31-401a-865e-6dc40fbff02b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358067 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.357884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-client-ca-bundle\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358067 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.357952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc2ec951-5b31-401a-865e-6dc40fbff02b-metrics-server-audit-profiles\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358067 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.357992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrksx\" (UniqueName: \"kubernetes.io/projected/cc2ec951-5b31-401a-865e-6dc40fbff02b-kube-api-access-xrksx\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358067 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.358024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-secret-metrics-server-client-certs\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358067 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.358044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-secret-metrics-server-tls\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358308 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.358060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc2ec951-5b31-401a-865e-6dc40fbff02b-audit-log\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358685 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.358658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2ec951-5b31-401a-865e-6dc40fbff02b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.358806 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.358666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc2ec951-5b31-401a-865e-6dc40fbff02b-audit-log\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.359097 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.359042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc2ec951-5b31-401a-865e-6dc40fbff02b-metrics-server-audit-profiles\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.361205 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.361183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-secret-metrics-server-client-certs\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.361305 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.361289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-client-ca-bundle\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.367342 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.367319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc2ec951-5b31-401a-865e-6dc40fbff02b-secret-metrics-server-tls\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.369488 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.369464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrksx\" (UniqueName: \"kubernetes.io/projected/cc2ec951-5b31-401a-865e-6dc40fbff02b-kube-api-access-xrksx\") pod \"metrics-server-5fc4d6979c-j5tfg\" (UID: \"cc2ec951-5b31-401a-865e-6dc40fbff02b\") " pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:01.478569 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.478529 2580 generic.go:356] "Generic (PLEG): container finished" podID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerID="cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e" exitCode=0 Jul 24 15:49:01.479081 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.478621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e"} Jul 24 15:49:01.550911 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:01.550872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:02.015502 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.015478 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fc4d6979c-j5tfg"] Jul 24 15:49:02.117842 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:49:02.117802 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2ec951_5b31_401a_865e_6dc40fbff02b.slice/crio-278520450b8936457ff53081324278b972c2f2dfe702cc8c60c8aecfca3838d4 WatchSource:0}: Error finding container 278520450b8936457ff53081324278b972c2f2dfe702cc8c60c8aecfca3838d4: Status 404 returned error can't find the container with id 278520450b8936457ff53081324278b972c2f2dfe702cc8c60c8aecfca3838d4 Jul 24 15:49:02.134470 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.134420 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9cc584dff-957kg"] Jul 24 15:49:02.151168 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.151136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9cc584dff-957kg"] Jul 24 15:49:02.151313 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.151292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.153513 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.153493 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 15:49:02.153513 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.153504 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 15:49:02.153772 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.153558 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 15:49:02.153772 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.153577 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 15:49:02.153772 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.153507 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xm72x\"" Jul 24 15:49:02.153772 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.153560 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 15:49:02.158703 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.158684 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 15:49:02.166269 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzqp\" (UniqueName: \"kubernetes.io/projected/c3e5cc0c-3eaa-4418-afd3-82f861e55003-kube-api-access-hjzqp\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.166381 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-serving-certs-ca-bundle\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.166381 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-telemeter-client-tls\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.166503 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-federate-client-tls\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.166503 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-secret-telemeter-client\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.166582 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.166582 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.166684 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.166622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-metrics-client-ca\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268000 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.267975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-telemeter-client-tls\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268117 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.268040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-federate-client-tls\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268117 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.268074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-secret-telemeter-client\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268117 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.268109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268280 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.268177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268280 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.268207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-metrics-client-ca\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268280 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.268236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzqp\" (UniqueName: \"kubernetes.io/projected/c3e5cc0c-3eaa-4418-afd3-82f861e55003-kube-api-access-hjzqp\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.268280 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.268267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-serving-certs-ca-bundle\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.269053 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.269007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-serving-certs-ca-bundle\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.269830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.269768 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-metrics-client-ca\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.270713 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.270661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3e5cc0c-3eaa-4418-afd3-82f861e55003-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.271966 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.271905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-secret-telemeter-client\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.271966 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.271905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-telemeter-client-tls\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.272251 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.272226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-federate-client-tls\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.273689 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.273669 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3e5cc0c-3eaa-4418-afd3-82f861e55003-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.277353 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.277295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzqp\" (UniqueName: \"kubernetes.io/projected/c3e5cc0c-3eaa-4418-afd3-82f861e55003-kube-api-access-hjzqp\") pod \"telemeter-client-9cc584dff-957kg\" (UID: \"c3e5cc0c-3eaa-4418-afd3-82f861e55003\") " pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.463722 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.463696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" Jul 24 15:49:02.484714 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.484655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" event={"ID":"cc2ec951-5b31-401a-865e-6dc40fbff02b","Type":"ContainerStarted","Data":"278520450b8936457ff53081324278b972c2f2dfe702cc8c60c8aecfca3838d4"} Jul 24 15:49:02.487547 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.487521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" event={"ID":"301f70fa-048d-48e6-b818-bb9f3f2a6243","Type":"ContainerStarted","Data":"944e2da6e5c28f6e311010377882c7e1014f332f9ceb937557240ac6c3524d37"} Jul 24 15:49:02.487653 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.487584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" event={"ID":"301f70fa-048d-48e6-b818-bb9f3f2a6243","Type":"ContainerStarted","Data":"e85b173776300fdb7e969102bda38cc7649af4d539e350bd02fdcf9db42cef43"} Jul 24 15:49:02.487653 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.487610 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" event={"ID":"301f70fa-048d-48e6-b818-bb9f3f2a6243","Type":"ContainerStarted","Data":"bd54e9faaa52121a9d13a9d36d8d72e6855059ce6cadc5860e60d1f268ab7281"} Jul 24 15:49:02.625065 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:02.625026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9cc584dff-957kg"] Jul 24 15:49:02.628826 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:49:02.628785 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e5cc0c_3eaa_4418_afd3_82f861e55003.slice/crio-e305a51e44dc819503f617b7faba624434dfa9a91dda450e51071681f9e78fc9 WatchSource:0}: Error finding container e305a51e44dc819503f617b7faba624434dfa9a91dda450e51071681f9e78fc9: Status 404 returned error can't find the container with id e305a51e44dc819503f617b7faba624434dfa9a91dda450e51071681f9e78fc9 Jul 24 15:49:03.495369 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:03.495336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" event={"ID":"c3e5cc0c-3eaa-4418-afd3-82f861e55003","Type":"ContainerStarted","Data":"e305a51e44dc819503f617b7faba624434dfa9a91dda450e51071681f9e78fc9"} Jul 24 15:49:03.497738 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:03.497710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerStarted","Data":"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12"} Jul 24 15:49:04.506431 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.506391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerStarted","Data":"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3"} Jul 24 15:49:04.506892 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.506483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerStarted","Data":"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf"} Jul 24 15:49:04.506892 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.506526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerStarted","Data":"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05"} Jul 24 15:49:04.506892 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.506562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerStarted","Data":"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86"} Jul 24 15:49:04.508181 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.508153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" event={"ID":"cc2ec951-5b31-401a-865e-6dc40fbff02b","Type":"ContainerStarted","Data":"e4a76c62bfffaa89e7c14442f753fb01e13778c8ed32d814eb0c1ef4591d2472"} Jul 24 15:49:04.511335 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.511309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" event={"ID":"301f70fa-048d-48e6-b818-bb9f3f2a6243","Type":"ContainerStarted","Data":"2b3724fd719a4170ea63874bcd55772ebf0647fc560bbaaa86c7b70fd5e164b0"} Jul 24 15:49:04.511432 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.511381 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" event={"ID":"301f70fa-048d-48e6-b818-bb9f3f2a6243","Type":"ContainerStarted","Data":"828aeb1a6e1d3f67d66dae13d02a5103420c1d467a0a66697f3d11374653b424"} Jul 24 15:49:04.511432 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.511420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" event={"ID":"301f70fa-048d-48e6-b818-bb9f3f2a6243","Type":"ContainerStarted","Data":"38657c3d6fe2432c9ed1727e0e27a031d7019ea5a10da4b3107398db23f5b322"} Jul 24 15:49:04.511972 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.511952 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:49:04.525704 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.525422 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" podStartSLOduration=1.6033152259999999 podStartE2EDuration="3.525407053s" totalImagesPullingTime="1.922091827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:04.525031336 +0000 UTC m=+113.046639874" watchObservedRunningTime="2026-07-24 15:49:04.525407053 +0000 UTC m=+113.047015595" Jul 24 15:49:04.548111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:04.548023 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" podStartSLOduration=2.215009005 podStartE2EDuration="6.54800932s" totalImagesPullingTime="4.333000315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:04.546998099 +0000 UTC m=+113.068606640" watchObservedRunningTime="2026-07-24 15:49:04.54800932 +0000 UTC m=+113.069617860" Jul 24 15:49:05.517802 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:05.517757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" event={"ID":"c3e5cc0c-3eaa-4418-afd3-82f861e55003","Type":"ContainerStarted","Data":"dcc10c0d5745392b7df5415577e73684e2dada074c3b9db8cbe07c89b0b03ce0"} Jul 24 15:49:05.518398 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:05.517834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" event={"ID":"c3e5cc0c-3eaa-4418-afd3-82f861e55003","Type":"ContainerStarted","Data":"6c65dc4056fe3dbb5f5d037e6c378496278f6dbc661d0886b93a143498cec581"} Jul 24 15:49:05.518398 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:05.517869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" event={"ID":"c3e5cc0c-3eaa-4418-afd3-82f861e55003","Type":"ContainerStarted","Data":"9c0deada344f47285e1c6c2bcdb7195ad8789a48058201985294873e27161ec5"} Jul 24 15:49:05.521093 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:05.521068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerStarted","Data":"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a"} Jul 24 15:49:05.539230 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:05.539173 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9cc584dff-957kg" podStartSLOduration=1.362035654 podStartE2EDuration="3.539158098s" totalImagesPullingTime="2.177122444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:05.53790067 +0000 UTC m=+114.059509209" watchObservedRunningTime="2026-07-24 15:49:05.539158098 +0000 UTC m=+114.060766638" Jul 24 15:49:05.562153 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:05.562110 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.023505742 podStartE2EDuration="8.562097013s" totalImagesPullingTime="3.513383873s" totalInitContainerRuntime="25.207398ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:05.561248041 +0000 UTC m=+114.082856595" watchObservedRunningTime="2026-07-24 15:49:05.562097013 +0000 UTC m=+114.083705553" Jul 24 15:49:10.526873 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:10.526845 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7cb95767d6-jnzr9" Jul 24 15:49:21.551671 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:21.551620 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:21.552216 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:21.551697 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:21.738332 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:21.738295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:49:21.740882 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:21.740841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837b9cd-372b-4e0e-ab2a-ff6013c26aea-metrics-certs\") pod \"network-metrics-daemon-ngh44\" (UID: \"1837b9cd-372b-4e0e-ab2a-ff6013c26aea\") " pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:49:21.925236 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:21.925155 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k45mm\"" Jul 24 15:49:21.934175 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:21.934147 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngh44" Jul 24 15:49:22.108799 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:22.108751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ngh44"] Jul 24 15:49:22.111407 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:49:22.111383 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1837b9cd_372b_4e0e_ab2a_ff6013c26aea.slice/crio-d84b890aafe7fa861705647a0f7a26f2338294a5728cc7feef9804788adffbdd WatchSource:0}: Error finding container d84b890aafe7fa861705647a0f7a26f2338294a5728cc7feef9804788adffbdd: Status 404 returned error can't find the container with id d84b890aafe7fa861705647a0f7a26f2338294a5728cc7feef9804788adffbdd Jul 24 15:49:22.591989 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:22.591957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ngh44" event={"ID":"1837b9cd-372b-4e0e-ab2a-ff6013c26aea","Type":"ContainerStarted","Data":"d84b890aafe7fa861705647a0f7a26f2338294a5728cc7feef9804788adffbdd"} Jul 24 15:49:23.597494 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:23.597454 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ngh44" event={"ID":"1837b9cd-372b-4e0e-ab2a-ff6013c26aea","Type":"ContainerStarted","Data":"4234640b4d09b834331bb8bb97f11732d381f64fe4a04847ae4516acffaf14e8"} Jul 24 15:49:23.597868 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:23.597526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ngh44" event={"ID":"1837b9cd-372b-4e0e-ab2a-ff6013c26aea","Type":"ContainerStarted","Data":"86061a1065b03eef4d531ce446a4312902d7835303edd4ccae2376184a01e970"} Jul 24 15:49:23.616071 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:23.615978 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ngh44" podStartSLOduration=130.642630477 podStartE2EDuration="2m11.615961873s" totalImagesPullingTime="973.331396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:23.613862187 +0000 UTC m=+132.135470750" watchObservedRunningTime="2026-07-24 15:49:23.615961873 +0000 UTC m=+132.137570413" Jul 24 15:49:32.631951 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:32.631900 2580 generic.go:356] "Generic (PLEG): container finished" podID="563706c0-a505-4373-b987-923b16eb34da" containerID="1002b2f1c3b448a06aa98e678aa16c85bf14a812ae1591ac843a90bd6554e405" exitCode=0 Jul 24 15:49:32.632329 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:32.631982 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" event={"ID":"563706c0-a505-4373-b987-923b16eb34da","Type":"ContainerDied","Data":"1002b2f1c3b448a06aa98e678aa16c85bf14a812ae1591ac843a90bd6554e405"} Jul 24 15:49:32.632367 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:32.632330 2580 scope.go:117] "RemoveContainer" containerID="1002b2f1c3b448a06aa98e678aa16c85bf14a812ae1591ac843a90bd6554e405" Jul 24 15:49:33.641712 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:33.641672 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4980202-43a2-43ea-a4db-90e02a50ff32" containerID="608aab468f57e4253fbb5573da51063b6ea304c24328b6422404dc292cbf041c" exitCode=0 Jul 24 15:49:33.642214 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:33.641747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" event={"ID":"a4980202-43a2-43ea-a4db-90e02a50ff32","Type":"ContainerDied","Data":"608aab468f57e4253fbb5573da51063b6ea304c24328b6422404dc292cbf041c"} Jul 24 15:49:33.642214 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:33.642195 2580 scope.go:117] "RemoveContainer" containerID="608aab468f57e4253fbb5573da51063b6ea304c24328b6422404dc292cbf041c" Jul 24 15:49:33.643685 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:33.643662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq" event={"ID":"563706c0-a505-4373-b987-923b16eb34da","Type":"ContainerStarted","Data":"949973f5470ccd88a918cf19f363310e5a2f212b94d111ed61dd280633d349f6"} Jul 24 15:49:34.649571 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:34.649538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8bcld" event={"ID":"a4980202-43a2-43ea-a4db-90e02a50ff32","Type":"ContainerStarted","Data":"6f4c1afdad223e6055cebac13bdb8a4c1fbd5b5032a7b1b7731ec3bd9d69db0e"} Jul 24 15:49:41.556432 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:41.556401 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:49:41.560275 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:49:41.560255 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5fc4d6979c-j5tfg" Jul 24 15:50:17.309899 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.309812 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:50:17.310412 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.310141 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="alertmanager" containerID="cri-o://028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12" gracePeriod=120 Jul 24 15:50:17.310412 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.310179 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="prom-label-proxy" containerID="cri-o://38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a" gracePeriod=120 Jul 24 15:50:17.310412 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.310172 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="config-reloader" containerID="cri-o://f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86" gracePeriod=120 Jul 24 15:50:17.310412 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.310190 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy" containerID="cri-o://b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf" gracePeriod=120 Jul 24 15:50:17.310412 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.310210 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-metric" containerID="cri-o://f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3" gracePeriod=120 Jul 24 15:50:17.310412 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.310162 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-web" containerID="cri-o://fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05" gracePeriod=120 Jul 24 15:50:17.826775 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826745 2580 generic.go:356] "Generic (PLEG): container finished" podID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerID="38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a" exitCode=0 Jul 24 15:50:17.826775 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826770 2580 generic.go:356] "Generic (PLEG): container finished" podID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerID="b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf" exitCode=0 Jul 24 15:50:17.826775 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826777 2580 generic.go:356] "Generic (PLEG): container finished" podID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerID="f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86" exitCode=0 Jul 24 15:50:17.826775 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826784 2580 generic.go:356] "Generic (PLEG): container finished" podID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerID="028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12" exitCode=0 Jul 24 15:50:17.827079 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a"} Jul 24 15:50:17.827079 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf"} Jul 24 15:50:17.827079 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86"} Jul 24 15:50:17.827079 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:17.826956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12"} Jul 24 15:50:18.552807 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.552784 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.643041 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643015 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643049 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-tls-assets\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643069 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-config-volume\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643098 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-main-db\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643130 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-997xj\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-kube-api-access-997xj\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643162 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-cluster-tls-config\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643192 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643227 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-web-config\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643259 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643295 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-config-out\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643336 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643387 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-metrics-client-ca\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643497 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643422 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-main-tls\") pod \"0090adb5-2358-499a-bb2e-9416787e5a1c\" (UID: \"0090adb5-2358-499a-bb2e-9416787e5a1c\") " Jul 24 15:50:18.643764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643507 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:50:18.643764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.643727 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-main-db\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.644342 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.644285 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:50:18.644461 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.644428 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:50:18.646196 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.646115 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:50:18.646630 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.646600 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:50:18.647044 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.647006 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:50:18.647140 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.647102 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:50:18.647296 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.647274 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-kube-api-access-997xj" (OuterVolumeSpecName: "kube-api-access-997xj") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "kube-api-access-997xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:50:18.647352 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.647275 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-config-out" (OuterVolumeSpecName: "config-out") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:50:18.647870 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.647847 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:50:18.647973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.647912 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-config-volume" (OuterVolumeSpecName: "config-volume") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:50:18.650965 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.650843 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:50:18.658823 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.658799 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-web-config" (OuterVolumeSpecName: "web-config") pod "0090adb5-2358-499a-bb2e-9416787e5a1c" (UID: "0090adb5-2358-499a-bb2e-9416787e5a1c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:50:18.744527 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744500 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744541 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-tls-assets\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744552 2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-config-volume\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744562 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997xj\" (UniqueName: \"kubernetes.io/projected/0090adb5-2358-499a-bb2e-9416787e5a1c-kube-api-access-997xj\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744570 2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-cluster-tls-config\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744578 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744587 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-web-config\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744595 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744603 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0090adb5-2358-499a-bb2e-9416787e5a1c-config-out\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744611 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744618 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090adb5-2358-499a-bb2e-9416787e5a1c-metrics-client-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.744655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.744627 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0090adb5-2358-499a-bb2e-9416787e5a1c-secret-alertmanager-main-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:50:18.834200 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.834165 2580 generic.go:356] "Generic (PLEG): container finished" podID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerID="f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3" exitCode=0 Jul 24 15:50:18.834200 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.834193 2580 generic.go:356] "Generic (PLEG): container finished" podID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerID="fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05" exitCode=0 Jul 24 15:50:18.834369 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.834247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3"} Jul 24 15:50:18.834369 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.834266 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.834369 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.834341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05"} Jul 24 15:50:18.834457 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.834373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0090adb5-2358-499a-bb2e-9416787e5a1c","Type":"ContainerDied","Data":"a21583ea74ca8fc0f7164353a8978aa7bb81de7b2801d1324fe24ba2fa49fe9f"} Jul 24 15:50:18.834457 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.834413 2580 scope.go:117] "RemoveContainer" containerID="38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a" Jul 24 15:50:18.842400 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.842376 2580 scope.go:117] "RemoveContainer" containerID="f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3" Jul 24 15:50:18.850123 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.850073 2580 scope.go:117] "RemoveContainer" containerID="b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf" Jul 24 15:50:18.856742 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.856717 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:50:18.857602 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.857585 2580 scope.go:117] "RemoveContainer" containerID="fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05" Jul 24 15:50:18.860861 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.860841 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:50:18.865227 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.865203 2580 scope.go:117] "RemoveContainer" containerID="f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86" Jul 24 15:50:18.873668 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.873653 2580 scope.go:117] "RemoveContainer" containerID="028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12" Jul 24 15:50:18.880662 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.880646 2580 scope.go:117] "RemoveContainer" containerID="cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e" Jul 24 15:50:18.886963 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.886926 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:50:18.887233 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887221 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-web" Jul 24 15:50:18.887284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887235 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-web" Jul 24 15:50:18.887284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887244 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="prom-label-proxy" Jul 24 15:50:18.887284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887249 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="prom-label-proxy" Jul 24 15:50:18.887284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887262 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-metric" Jul 24 15:50:18.887284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887267 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-metric" Jul 24 15:50:18.887284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887274 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy" Jul 24 15:50:18.887284 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887279 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887288 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="alertmanager" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887293 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="alertmanager" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887300 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="init-config-reloader" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887305 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="init-config-reloader" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887311 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="config-reloader" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887315 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="config-reloader" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887358 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-web" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887364 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy-metric" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887371 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="kube-rbac-proxy" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887377 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="prom-label-proxy" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887384 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="alertmanager" Jul 24 15:50:18.887480 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.887390 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" containerName="config-reloader" Jul 24 15:50:18.890298 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.890282 2580 scope.go:117] "RemoveContainer" containerID="38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a" Jul 24 15:50:18.890658 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:50:18.890635 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a\": container with ID starting with 38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a not found: ID does not exist" containerID="38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a" Jul 24 15:50:18.890726 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.890670 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a"} err="failed to get container status \"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a\": rpc error: code = NotFound desc = could not find container \"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a\": container with ID starting with 38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a not found: ID does not exist" Jul 24 15:50:18.890726 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.890708 2580 scope.go:117] "RemoveContainer" containerID="f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3" Jul 24 15:50:18.891049 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:50:18.891030 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3\": container with ID starting with f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3 not found: ID does not exist" containerID="f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3" Jul 24 15:50:18.891097 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.891056 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3"} err="failed to get container status \"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3\": rpc error: code = NotFound desc = could not find container \"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3\": container with ID starting with f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3 not found: ID does not exist" Jul 24 15:50:18.891097 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.891074 2580 scope.go:117] "RemoveContainer" containerID="b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf" Jul 24 15:50:18.891398 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:50:18.891381 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf\": container with ID starting with b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf not found: ID does not exist" containerID="b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf" Jul 24 15:50:18.891455 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.891404 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf"} err="failed to get container status \"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf\": rpc error: code = NotFound desc = could not find container \"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf\": container with ID starting with b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf not found: ID does not exist" Jul 24 15:50:18.891455 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.891418 2580 scope.go:117] "RemoveContainer" containerID="fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05" Jul 24 15:50:18.891645 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:50:18.891629 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05\": container with ID starting with fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05 not found: ID does not exist" containerID="fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05" Jul 24 15:50:18.891688 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.891648 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05"} err="failed to get container status \"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05\": rpc error: code = NotFound desc = could not find container \"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05\": container with ID starting with fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05 not found: ID does not exist" Jul 24 15:50:18.891688 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.891660 2580 scope.go:117] "RemoveContainer" containerID="f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86" Jul 24 15:50:18.892056 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:50:18.892040 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86\": container with ID starting with f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86 not found: ID does not exist" containerID="f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86" Jul 24 15:50:18.892106 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892062 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86"} err="failed to get container status \"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86\": rpc error: code = NotFound desc = could not find container \"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86\": container with ID starting with f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86 not found: ID does not exist" Jul 24 15:50:18.892106 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892075 2580 scope.go:117] "RemoveContainer" containerID="028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12" Jul 24 15:50:18.892233 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:50:18.892218 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12\": container with ID starting with 028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12 not found: ID does not exist" containerID="028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12" Jul 24 15:50:18.892272 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892236 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12"} err="failed to get container status \"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12\": rpc error: code = NotFound desc = could not find container \"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12\": container with ID starting with 028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12 not found: ID does not exist" Jul 24 15:50:18.892272 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892247 2580 scope.go:117] "RemoveContainer" containerID="cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e" Jul 24 15:50:18.892401 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.892440 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:50:18.892404 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e\": container with ID starting with cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e not found: ID does not exist" containerID="cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e" Jul 24 15:50:18.892440 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892418 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e"} err="failed to get container status \"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e\": rpc error: code = NotFound desc = could not find container \"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e\": container with ID starting with cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e not found: ID does not exist" Jul 24 15:50:18.892440 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892428 2580 scope.go:117] "RemoveContainer" containerID="38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a" Jul 24 15:50:18.892645 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892616 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a"} err="failed to get container status \"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a\": rpc error: code = NotFound desc = could not find container \"38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a\": container with ID starting with 38697302ca29ea65d873c2a4da2a1db6bee82a540206815b382c89799cb4b63a not found: ID does not exist" Jul 24 15:50:18.892701 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892648 2580 scope.go:117] "RemoveContainer" containerID="f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3" Jul 24 15:50:18.893047 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892922 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3"} err="failed to get container status \"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3\": rpc error: code = NotFound desc = could not find container \"f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3\": container with ID starting with f8e9da90e59744de97935951e29bc6b4bab7973d27e21f901eac703e5d9cf7f3 not found: ID does not exist" Jul 24 15:50:18.893047 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.892973 2580 scope.go:117] "RemoveContainer" containerID="b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf" Jul 24 15:50:18.893274 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893255 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf"} err="failed to get container status \"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf\": rpc error: code = NotFound desc = could not find container \"b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf\": container with ID starting with b392e8929add026d7043f9e35baf915d6cecdc80f3ff8ebe574f356e299de5cf not found: ID does not exist" Jul 24 15:50:18.893327 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893276 2580 scope.go:117] "RemoveContainer" containerID="fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05" Jul 24 15:50:18.893513 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893489 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05"} err="failed to get container status \"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05\": rpc error: code = NotFound desc = could not find container \"fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05\": container with ID starting with fc33128876600bd60a19852b9ad2296a4baa94aac2d13a3274b159c64ae1ab05 not found: ID does not exist" Jul 24 15:50:18.893513 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893513 2580 scope.go:117] "RemoveContainer" containerID="f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86" Jul 24 15:50:18.893729 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893713 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86"} err="failed to get container status \"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86\": rpc error: code = NotFound desc = could not find container \"f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86\": container with ID starting with f52adc984552fd65579598fe17769c237b4090185447d491cccac77b2c655b86 not found: ID does not exist" Jul 24 15:50:18.893770 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893731 2580 scope.go:117] "RemoveContainer" containerID="028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12" Jul 24 15:50:18.893977 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893960 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12"} err="failed to get container status \"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12\": rpc error: code = NotFound desc = could not find container \"028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12\": container with ID starting with 028e96e311bf2786cdda012e0bbca3c362df3af14e9739efaf8d77ec89d05a12 not found: ID does not exist" Jul 24 15:50:18.893977 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.893977 2580 scope.go:117] "RemoveContainer" containerID="cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e" Jul 24 15:50:18.894191 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.894174 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e"} err="failed to get container status \"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e\": rpc error: code = NotFound desc = could not find container \"cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e\": container with ID starting with cbc067904dd49eb8bdb91d15b3954cc06265ff68902730ddac4527856e91f39e not found: ID does not exist" Jul 24 15:50:18.894706 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.894686 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 15:50:18.894970 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.894919 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 15:50:18.894970 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.894958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 15:50:18.895112 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.894962 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fslvz\"" Jul 24 15:50:18.895112 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.895058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 15:50:18.895196 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.895145 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 15:50:18.895251 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.895236 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 15:50:18.895294 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.895283 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 15:50:18.895333 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.895291 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 15:50:18.900207 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.900189 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 15:50:18.904019 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.903999 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:50:18.946088 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3d31bc6-c37b-4fcb-aa38-840ce2344737-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946216 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946216 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946216 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3d31bc6-c37b-4fcb-aa38-840ce2344737-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946352 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-web-config\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946352 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d31bc6-c37b-4fcb-aa38-840ce2344737-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946352 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946352 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946352 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2t5\" (UniqueName: \"kubernetes.io/projected/f3d31bc6-c37b-4fcb-aa38-840ce2344737-kube-api-access-fs2t5\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946495 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3d31bc6-c37b-4fcb-aa38-840ce2344737-config-out\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946495 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946495 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:18.946495 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:18.946449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d31bc6-c37b-4fcb-aa38-840ce2344737-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047282 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047421 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3d31bc6-c37b-4fcb-aa38-840ce2344737-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047421 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-web-config\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047421 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d31bc6-c37b-4fcb-aa38-840ce2344737-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047421 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047421 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047421 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2t5\" (UniqueName: \"kubernetes.io/projected/f3d31bc6-c37b-4fcb-aa38-840ce2344737-kube-api-access-fs2t5\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047714 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3d31bc6-c37b-4fcb-aa38-840ce2344737-config-out\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047714 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047714 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047714 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d31bc6-c37b-4fcb-aa38-840ce2344737-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047959 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3d31bc6-c37b-4fcb-aa38-840ce2344737-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047959 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.047959 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.047838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3d31bc6-c37b-4fcb-aa38-840ce2344737-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.048517 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.048402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d31bc6-c37b-4fcb-aa38-840ce2344737-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.048801 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.048773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d31bc6-c37b-4fcb-aa38-840ce2344737-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.050904 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.050712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.050904 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.050838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.050904 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.050865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-web-config\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.051098 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.051000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.051144 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.051126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.051340 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.051317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3d31bc6-c37b-4fcb-aa38-840ce2344737-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.051382 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.051358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3d31bc6-c37b-4fcb-aa38-840ce2344737-config-out\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.051834 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.051807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.052262 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.052246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3d31bc6-c37b-4fcb-aa38-840ce2344737-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.056495 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.056476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2t5\" (UniqueName: \"kubernetes.io/projected/f3d31bc6-c37b-4fcb-aa38-840ce2344737-kube-api-access-fs2t5\") pod \"alertmanager-main-0\" (UID: \"f3d31bc6-c37b-4fcb-aa38-840ce2344737\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.201832 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.201749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:50:19.336453 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.336428 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:50:19.338913 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:50:19.338886 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d31bc6_c37b_4fcb_aa38_840ce2344737.slice/crio-aed4a4551e8b99ea225bb9959dcb2f4afd60252ee7a06aaebd161f32bb3dbace WatchSource:0}: Error finding container aed4a4551e8b99ea225bb9959dcb2f4afd60252ee7a06aaebd161f32bb3dbace: Status 404 returned error can't find the container with id aed4a4551e8b99ea225bb9959dcb2f4afd60252ee7a06aaebd161f32bb3dbace Jul 24 15:50:19.840137 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.840098 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3d31bc6-c37b-4fcb-aa38-840ce2344737" containerID="ae606e4cc72a278083ab16035a31af62af18a0cbf4e883801ec4e088c932e097" exitCode=0 Jul 24 15:50:19.840600 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.840191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerDied","Data":"ae606e4cc72a278083ab16035a31af62af18a0cbf4e883801ec4e088c932e097"} Jul 24 15:50:19.840600 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:19.840251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerStarted","Data":"aed4a4551e8b99ea225bb9959dcb2f4afd60252ee7a06aaebd161f32bb3dbace"} Jul 24 15:50:20.009631 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.009604 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0090adb5-2358-499a-bb2e-9416787e5a1c" path="/var/lib/kubelet/pods/0090adb5-2358-499a-bb2e-9416787e5a1c/volumes" Jul 24 15:50:20.848393 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.848357 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerStarted","Data":"ab42d172651766c8ca9238117683070a9cfbaf5f9973d232ce852a5c5a100258"} Jul 24 15:50:20.848830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.848416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerStarted","Data":"686dc09d823034376d66ab2e20d78b764b2bded56359ae3d04ef246db6af1edc"} Jul 24 15:50:20.848830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.848442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerStarted","Data":"8244580e7748dee855786f2e6f5aecfa6eb4015695473a700023bed2fa2fcfc5"} Jul 24 15:50:20.848830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.848472 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerStarted","Data":"7c6d7ab3962ce98bb2a3946a51c181e4258f68cb63fbbbf25b59172869757d87"} Jul 24 15:50:20.848830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.848497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerStarted","Data":"cf779c920b1d92ed27f289b7412c2beda2012e9306f822e659532ce58cf0c9b6"} Jul 24 15:50:20.848830 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.848520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3d31bc6-c37b-4fcb-aa38-840ce2344737","Type":"ContainerStarted","Data":"339d087c2bea9148c01b490fbb989b0bf4c5f106ba3b645c4a2be4dfa5c57b4b"} Jul 24 15:50:20.873067 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:50:20.873023 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.855041737 podStartE2EDuration="2.873006158s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.964421ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:50:20.872310946 +0000 UTC m=+189.393919507" watchObservedRunningTime="2026-07-24 15:50:20.873006158 +0000 UTC m=+189.394614697" Jul 24 15:52:07.778921 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.778893 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7"] Jul 24 15:52:07.782141 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.782126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:07.784156 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.784136 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 15:52:07.784616 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.784599 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 15:52:07.784753 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.784621 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x2lfz\"" Jul 24 15:52:07.791193 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.791166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7"] Jul 24 15:52:07.848944 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.848919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f545ff89-d2d7-4284-9062-3f5fc28f8b54-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jnkb7\" (UID: \"f545ff89-d2d7-4284-9062-3f5fc28f8b54\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:07.849035 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.848970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-747nz\" (UniqueName: \"kubernetes.io/projected/f545ff89-d2d7-4284-9062-3f5fc28f8b54-kube-api-access-747nz\") pod \"cert-manager-cainjector-559cdd8c87-jnkb7\" (UID: \"f545ff89-d2d7-4284-9062-3f5fc28f8b54\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:07.949806 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.949777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747nz\" (UniqueName: \"kubernetes.io/projected/f545ff89-d2d7-4284-9062-3f5fc28f8b54-kube-api-access-747nz\") pod \"cert-manager-cainjector-559cdd8c87-jnkb7\" (UID: \"f545ff89-d2d7-4284-9062-3f5fc28f8b54\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:07.949903 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.949842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f545ff89-d2d7-4284-9062-3f5fc28f8b54-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jnkb7\" (UID: \"f545ff89-d2d7-4284-9062-3f5fc28f8b54\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:07.958115 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.958085 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f545ff89-d2d7-4284-9062-3f5fc28f8b54-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jnkb7\" (UID: \"f545ff89-d2d7-4284-9062-3f5fc28f8b54\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:07.958206 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:07.958157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-747nz\" (UniqueName: \"kubernetes.io/projected/f545ff89-d2d7-4284-9062-3f5fc28f8b54-kube-api-access-747nz\") pod \"cert-manager-cainjector-559cdd8c87-jnkb7\" (UID: \"f545ff89-d2d7-4284-9062-3f5fc28f8b54\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:08.104301 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:08.104229 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" Jul 24 15:52:08.235450 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:08.235427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7"] Jul 24 15:52:08.238308 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:52:08.238285 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf545ff89_d2d7_4284_9062_3f5fc28f8b54.slice/crio-a8c7e51e6501b04eb9093c6fad0c1895afd6d618d33b90408e03d654efb854be WatchSource:0}: Error finding container a8c7e51e6501b04eb9093c6fad0c1895afd6d618d33b90408e03d654efb854be: Status 404 returned error can't find the container with id a8c7e51e6501b04eb9093c6fad0c1895afd6d618d33b90408e03d654efb854be Jul 24 15:52:08.268401 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:08.268373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" event={"ID":"f545ff89-d2d7-4284-9062-3f5fc28f8b54","Type":"ContainerStarted","Data":"a8c7e51e6501b04eb9093c6fad0c1895afd6d618d33b90408e03d654efb854be"} Jul 24 15:52:11.895264 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:11.895237 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 15:52:11.902531 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:11.902507 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 15:52:11.906303 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:11.906280 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 15:52:12.286236 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:12.286206 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" event={"ID":"f545ff89-d2d7-4284-9062-3f5fc28f8b54","Type":"ContainerStarted","Data":"adc6e9bb176b591070fed10dd6c605fbd4ef1eeff3348ebaa9536c8541fc5fe3"} Jul 24 15:52:12.300875 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:12.300827 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jnkb7" podStartSLOduration=2.146681239 podStartE2EDuration="5.300813314s" totalImagesPullingTime="3.154132075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:12.299896096 +0000 UTC m=+300.821504648" watchObservedRunningTime="2026-07-24 15:52:12.300813314 +0000 UTC m=+300.822421854" Jul 24 15:52:47.279318 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.279283 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-777c588886-r2r6f"] Jul 24 15:52:47.287665 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.287641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.289655 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.289628 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 15:52:47.290269 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.290236 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 15:52:47.290269 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.290241 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 15:52:47.290424 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.290273 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 15:52:47.290424 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.290241 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:52:47.290424 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.290292 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ftnp6\"" Jul 24 15:52:47.294912 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.294691 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-777c588886-r2r6f"] Jul 24 15:52:47.461890 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.461851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfc57b0f-baff-4518-a684-2d59590fb369-manager-config\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.462075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.461899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j6ms\" (UniqueName: \"kubernetes.io/projected/cfc57b0f-baff-4518-a684-2d59590fb369-kube-api-access-5j6ms\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.462075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.461973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfc57b0f-baff-4518-a684-2d59590fb369-cert\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.462075 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.462031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc57b0f-baff-4518-a684-2d59590fb369-metrics-cert\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.563212 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.563142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j6ms\" (UniqueName: \"kubernetes.io/projected/cfc57b0f-baff-4518-a684-2d59590fb369-kube-api-access-5j6ms\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.563212 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.563182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfc57b0f-baff-4518-a684-2d59590fb369-cert\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.563212 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.563212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc57b0f-baff-4518-a684-2d59590fb369-metrics-cert\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.563443 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.563300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfc57b0f-baff-4518-a684-2d59590fb369-manager-config\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.564078 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.563989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfc57b0f-baff-4518-a684-2d59590fb369-manager-config\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.565745 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.565715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfc57b0f-baff-4518-a684-2d59590fb369-cert\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.565829 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.565763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc57b0f-baff-4518-a684-2d59590fb369-metrics-cert\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.571760 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.571741 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j6ms\" (UniqueName: \"kubernetes.io/projected/cfc57b0f-baff-4518-a684-2d59590fb369-kube-api-access-5j6ms\") pod \"lws-controller-manager-777c588886-r2r6f\" (UID: \"cfc57b0f-baff-4518-a684-2d59590fb369\") " pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.597587 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.597562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:47.729631 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.729611 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-777c588886-r2r6f"] Jul 24 15:52:47.732437 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:52:47.732410 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc57b0f_baff_4518_a684_2d59590fb369.slice/crio-d42c6111eba06c127c95f851c6b17da0052f8410b11d05f71de25d59d8c8a1f3 WatchSource:0}: Error finding container d42c6111eba06c127c95f851c6b17da0052f8410b11d05f71de25d59d8c8a1f3: Status 404 returned error can't find the container with id d42c6111eba06c127c95f851c6b17da0052f8410b11d05f71de25d59d8c8a1f3 Jul 24 15:52:47.734423 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:47.734408 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:52:48.429077 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:48.429035 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" event={"ID":"cfc57b0f-baff-4518-a684-2d59590fb369","Type":"ContainerStarted","Data":"d42c6111eba06c127c95f851c6b17da0052f8410b11d05f71de25d59d8c8a1f3"} Jul 24 15:52:50.439834 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:50.439793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" event={"ID":"cfc57b0f-baff-4518-a684-2d59590fb369","Type":"ContainerStarted","Data":"34dff23924e701f54c6697cd6af4767639712b04bc94aa505ad55901aaff002e"} Jul 24 15:52:50.440332 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:50.439917 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:52:50.454851 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:50.454805 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" podStartSLOduration=1.210936407 podStartE2EDuration="3.45479121s" totalImagesPullingTime="2.243854803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:50.453725256 +0000 UTC m=+338.975333795" watchObservedRunningTime="2026-07-24 15:52:50.45479121 +0000 UTC m=+338.976399748" Jul 24 15:52:58.303396 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.303369 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q"] Jul 24 15:52:58.308199 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.308171 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.317592 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.317548 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 15:52:58.317746 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.317657 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9glrs\"" Jul 24 15:52:58.317746 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.317696 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 15:52:58.317894 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.317881 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 15:52:58.318650 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.318633 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 15:52:58.332496 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.332476 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q"] Jul 24 15:52:58.343605 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.343573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6kl\" (UniqueName: \"kubernetes.io/projected/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-kube-api-access-5h6kl\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.343725 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.343623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-manifests\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.343789 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.343724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-webhook-cert\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.343789 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.343764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-tmp\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.343894 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.343794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.444832 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.444813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6kl\" (UniqueName: \"kubernetes.io/projected/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-kube-api-access-5h6kl\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.444960 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.444846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-manifests\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.444960 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.444869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-webhook-cert\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.444960 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.444898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-tmp\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.445078 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.444963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.445297 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.445271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-manifests\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.447357 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.447333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-tmp\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.447463 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.447448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.447673 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.447651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-webhook-cert\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.452724 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.452696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6kl\" (UniqueName: \"kubernetes.io/projected/7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3-kube-api-access-5h6kl\") pod \"opendatahub-operator-controller-manager-749946f7d7-swn4q\" (UID: \"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3\") " pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.618879 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.618803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:52:58.749871 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:58.749843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q"] Jul 24 15:52:58.752400 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:52:58.752373 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fed1aca_09f7_4a1f_aaf6_ea679bf42fc3.slice/crio-66d83dd15e1f7e22c2bae79707facffd718b06d778e69f388ebc93eb8c70acdd WatchSource:0}: Error finding container 66d83dd15e1f7e22c2bae79707facffd718b06d778e69f388ebc93eb8c70acdd: Status 404 returned error can't find the container with id 66d83dd15e1f7e22c2bae79707facffd718b06d778e69f388ebc93eb8c70acdd Jul 24 15:52:59.484367 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:52:59.484293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" event={"ID":"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3","Type":"ContainerStarted","Data":"66d83dd15e1f7e22c2bae79707facffd718b06d778e69f388ebc93eb8c70acdd"} Jul 24 15:53:01.452423 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:01.452383 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-777c588886-r2r6f" Jul 24 15:53:02.500846 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:02.500817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" event={"ID":"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3","Type":"ContainerStarted","Data":"d88c2787ef23d8325706c9bc6fdc6b0c49dcb1a10b9c1b4c183540cb49674b4f"} Jul 24 15:53:02.762611 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:53:02.762583 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fed1aca_09f7_4a1f_aaf6_ea679bf42fc3.slice/crio-conmon-d88c2787ef23d8325706c9bc6fdc6b0c49dcb1a10b9c1b4c183540cb49674b4f.scope\": RecentStats: unable to find data in memory cache]" Jul 24 15:53:03.506721 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:03.506690 2580 generic.go:356] "Generic (PLEG): container finished" podID="7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3" containerID="d88c2787ef23d8325706c9bc6fdc6b0c49dcb1a10b9c1b4c183540cb49674b4f" exitCode=0 Jul 24 15:53:03.507099 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:03.506757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" event={"ID":"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3","Type":"ContainerDied","Data":"d88c2787ef23d8325706c9bc6fdc6b0c49dcb1a10b9c1b4c183540cb49674b4f"} Jul 24 15:53:04.513210 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:04.513175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" event={"ID":"7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3","Type":"ContainerStarted","Data":"27e4fb76ed8593894b5fe9d9e6965eae2939c3013913d6fc995cd5c719889090"} Jul 24 15:53:04.513660 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:04.513257 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:53:04.533397 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:04.533346 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" podStartSLOduration=2.21258 podStartE2EDuration="6.533329086s" totalImagesPullingTime="3.25310624s" totalInitContainerRuntime="1.067642846s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:04.532230936 +0000 UTC m=+353.053839475" watchObservedRunningTime="2026-07-24 15:53:04.533329086 +0000 UTC m=+353.054937627" Jul 24 15:53:15.518904 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:15.518875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-749946f7d7-swn4q" Jul 24 15:53:23.217450 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.217412 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-545786b4d4-v222p"] Jul 24 15:53:23.229781 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.229757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545786b4d4-v222p"] Jul 24 15:53:23.229986 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.229846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.232209 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.232181 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 15:53:23.232345 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.232221 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 15:53:23.232345 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.232239 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 15:53:23.232752 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.232726 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9blfk\"" Jul 24 15:53:23.232890 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.232773 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 15:53:23.360608 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.360574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3cd4577-c073-49d2-ad94-70f3ace80f8e-tmp\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.360764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.360644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmbk\" (UniqueName: \"kubernetes.io/projected/f3cd4577-c073-49d2-ad94-70f3ace80f8e-kube-api-access-fnmbk\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.360764 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.360707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd4577-c073-49d2-ad94-70f3ace80f8e-tls-certs\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.461883 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.461848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmbk\" (UniqueName: \"kubernetes.io/projected/f3cd4577-c073-49d2-ad94-70f3ace80f8e-kube-api-access-fnmbk\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.461883 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.461899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd4577-c073-49d2-ad94-70f3ace80f8e-tls-certs\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.462105 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.461957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3cd4577-c073-49d2-ad94-70f3ace80f8e-tmp\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.464374 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.464346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3cd4577-c073-49d2-ad94-70f3ace80f8e-tmp\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.464703 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.464640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cd4577-c073-49d2-ad94-70f3ace80f8e-tls-certs\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.470873 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.470819 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmbk\" (UniqueName: \"kubernetes.io/projected/f3cd4577-c073-49d2-ad94-70f3ace80f8e-kube-api-access-fnmbk\") pod \"kube-auth-proxy-545786b4d4-v222p\" (UID: \"f3cd4577-c073-49d2-ad94-70f3ace80f8e\") " pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.540629 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.540587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" Jul 24 15:53:23.678591 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:23.678567 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545786b4d4-v222p"] Jul 24 15:53:23.680412 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:53:23.680382 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cd4577_c073_49d2_ad94_70f3ace80f8e.slice/crio-11676019b32a8e32309ef79a928eeb02109450d4f7a5a3b3c5d7b9850972ec37 WatchSource:0}: Error finding container 11676019b32a8e32309ef79a928eeb02109450d4f7a5a3b3c5d7b9850972ec37: Status 404 returned error can't find the container with id 11676019b32a8e32309ef79a928eeb02109450d4f7a5a3b3c5d7b9850972ec37 Jul 24 15:53:24.600625 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:24.600588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" event={"ID":"f3cd4577-c073-49d2-ad94-70f3ace80f8e","Type":"ContainerStarted","Data":"11676019b32a8e32309ef79a928eeb02109450d4f7a5a3b3c5d7b9850972ec37"} Jul 24 15:53:26.612020 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:53:26.611978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" event={"ID":"f3cd4577-c073-49d2-ad94-70f3ace80f8e","Type":"ContainerStarted","Data":"519016fa8f2a7951a6082df928a2f1e360c50f1271518f639f62f1df941b3232"} Jul 24 15:54:57.295697 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.295635 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-545786b4d4-v222p" podStartSLOduration=92.102383176 podStartE2EDuration="1m34.295617994s" totalImagesPullingTime="2.193234818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:26.650901475 +0000 UTC m=+375.172510016" watchObservedRunningTime="2026-07-24 15:54:57.295617994 +0000 UTC m=+465.817226533" Jul 24 15:54:57.296252 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.295918 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b7296"] Jul 24 15:54:57.299469 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.299449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.301503 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.301481 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 15:54:57.301622 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.301539 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:54:57.302017 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.301997 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mgtkk\"" Jul 24 15:54:57.302169 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.302015 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:54:57.302169 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.302047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 15:54:57.309133 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.309113 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b7296"] Jul 24 15:54:57.341710 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.341685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8dq\" (UniqueName: \"kubernetes.io/projected/95bd019a-375b-44e2-84d9-0a306f8672a7-kube-api-access-nv8dq\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.341833 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.341716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95bd019a-375b-44e2-84d9-0a306f8672a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.341833 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.341756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95bd019a-375b-44e2-84d9-0a306f8672a7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.442763 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.442734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8dq\" (UniqueName: \"kubernetes.io/projected/95bd019a-375b-44e2-84d9-0a306f8672a7-kube-api-access-nv8dq\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.442763 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.442762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95bd019a-375b-44e2-84d9-0a306f8672a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.442962 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.442795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95bd019a-375b-44e2-84d9-0a306f8672a7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.443414 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.443397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95bd019a-375b-44e2-84d9-0a306f8672a7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.445201 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.445184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95bd019a-375b-44e2-84d9-0a306f8672a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.455395 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.455371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8dq\" (UniqueName: \"kubernetes.io/projected/95bd019a-375b-44e2-84d9-0a306f8672a7-kube-api-access-nv8dq\") pod \"kuadrant-console-plugin-66b5774985-b7296\" (UID: \"95bd019a-375b-44e2-84d9-0a306f8672a7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.611098 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.611029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" Jul 24 15:54:57.743867 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:57.743844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b7296"] Jul 24 15:54:57.746527 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:54:57.746499 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95bd019a_375b_44e2_84d9_0a306f8672a7.slice/crio-d4965ba0f54f8e83d95c8bb3df2891a28af845a00ff9abbf629603b3e07db97c WatchSource:0}: Error finding container d4965ba0f54f8e83d95c8bb3df2891a28af845a00ff9abbf629603b3e07db97c: Status 404 returned error can't find the container with id d4965ba0f54f8e83d95c8bb3df2891a28af845a00ff9abbf629603b3e07db97c Jul 24 15:54:58.009382 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:54:58.009353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" event={"ID":"95bd019a-375b-44e2-84d9-0a306f8672a7","Type":"ContainerStarted","Data":"d4965ba0f54f8e83d95c8bb3df2891a28af845a00ff9abbf629603b3e07db97c"} Jul 24 15:55:19.111853 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:19.111821 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" event={"ID":"95bd019a-375b-44e2-84d9-0a306f8672a7","Type":"ContainerStarted","Data":"27413fb6fe8b2faffdc95660098f2ab6eb2302b2eaaf93f5eb784e9150844c23"} Jul 24 15:55:19.126128 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:19.126070 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b7296" podStartSLOduration=1.363136632 podStartE2EDuration="22.126051731s" totalImagesPullingTime="20.762915099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:19.125182006 +0000 UTC m=+487.646790570" watchObservedRunningTime="2026-07-24 15:55:19.126051731 +0000 UTC m=+487.647660272" Jul 24 15:55:39.507247 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.507212 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8m7g8"] Jul 24 15:55:39.509483 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.509463 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.511482 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.511466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 15:55:39.518755 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.518731 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8m7g8"] Jul 24 15:55:39.602880 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.602845 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8m7g8"] Jul 24 15:55:39.623350 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.623323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72536052-6452-4d62-8d3e-2f6ad68f6fe4-config-file\") pod \"limitador-limitador-64c94fd64f-8m7g8\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.623472 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.623421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdnw\" (UniqueName: \"kubernetes.io/projected/72536052-6452-4d62-8d3e-2f6ad68f6fe4-kube-api-access-cfdnw\") pod \"limitador-limitador-64c94fd64f-8m7g8\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.724451 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.724417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdnw\" (UniqueName: \"kubernetes.io/projected/72536052-6452-4d62-8d3e-2f6ad68f6fe4-kube-api-access-cfdnw\") pod \"limitador-limitador-64c94fd64f-8m7g8\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.724622 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.724473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72536052-6452-4d62-8d3e-2f6ad68f6fe4-config-file\") pod \"limitador-limitador-64c94fd64f-8m7g8\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.725118 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.725100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72536052-6452-4d62-8d3e-2f6ad68f6fe4-config-file\") pod \"limitador-limitador-64c94fd64f-8m7g8\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.731699 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.731665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdnw\" (UniqueName: \"kubernetes.io/projected/72536052-6452-4d62-8d3e-2f6ad68f6fe4-kube-api-access-cfdnw\") pod \"limitador-limitador-64c94fd64f-8m7g8\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.820154 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.820069 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:39.952725 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:39.952672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8m7g8"] Jul 24 15:55:39.954996 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:55:39.954969 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72536052_6452_4d62_8d3e_2f6ad68f6fe4.slice/crio-4ea59a0bb369e231faaeb278d5339c5b56fb3731e7cd281e6636df6243aa3444 WatchSource:0}: Error finding container 4ea59a0bb369e231faaeb278d5339c5b56fb3731e7cd281e6636df6243aa3444: Status 404 returned error can't find the container with id 4ea59a0bb369e231faaeb278d5339c5b56fb3731e7cd281e6636df6243aa3444 Jul 24 15:55:40.202004 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.201901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" event={"ID":"72536052-6452-4d62-8d3e-2f6ad68f6fe4","Type":"ContainerStarted","Data":"4ea59a0bb369e231faaeb278d5339c5b56fb3731e7cd281e6636df6243aa3444"} Jul 24 15:55:40.306585 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.306546 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-478jl"] Jul 24 15:55:40.310009 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.309983 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-478jl" Jul 24 15:55:40.312380 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.312356 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kwhg4\"" Jul 24 15:55:40.317079 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.317059 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-478jl"] Jul 24 15:55:40.432305 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.432273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bh2p\" (UniqueName: \"kubernetes.io/projected/aa844b98-b5b9-4ea0-a4bf-bb553edbec2d-kube-api-access-4bh2p\") pod \"authorino-58cd955675-478jl\" (UID: \"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d\") " pod="kuadrant-system/authorino-58cd955675-478jl" Jul 24 15:55:40.533085 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.533053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bh2p\" (UniqueName: \"kubernetes.io/projected/aa844b98-b5b9-4ea0-a4bf-bb553edbec2d-kube-api-access-4bh2p\") pod \"authorino-58cd955675-478jl\" (UID: \"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d\") " pod="kuadrant-system/authorino-58cd955675-478jl" Jul 24 15:55:40.546107 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.546080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bh2p\" (UniqueName: \"kubernetes.io/projected/aa844b98-b5b9-4ea0-a4bf-bb553edbec2d-kube-api-access-4bh2p\") pod \"authorino-58cd955675-478jl\" (UID: \"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d\") " pod="kuadrant-system/authorino-58cd955675-478jl" Jul 24 15:55:40.621047 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.621004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-478jl" Jul 24 15:55:40.795668 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:40.795638 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-478jl"] Jul 24 15:55:40.798784 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:55:40.798754 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa844b98_b5b9_4ea0_a4bf_bb553edbec2d.slice/crio-f1ba635e99f88904137ecd03fa4d4460ebb9bbc7b81120498348183ebc57cdc7 WatchSource:0}: Error finding container f1ba635e99f88904137ecd03fa4d4460ebb9bbc7b81120498348183ebc57cdc7: Status 404 returned error can't find the container with id f1ba635e99f88904137ecd03fa4d4460ebb9bbc7b81120498348183ebc57cdc7 Jul 24 15:55:41.208515 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:41.208426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-478jl" event={"ID":"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d","Type":"ContainerStarted","Data":"f1ba635e99f88904137ecd03fa4d4460ebb9bbc7b81120498348183ebc57cdc7"} Jul 24 15:55:43.223163 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:43.223082 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" event={"ID":"72536052-6452-4d62-8d3e-2f6ad68f6fe4","Type":"ContainerStarted","Data":"72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a"} Jul 24 15:55:43.223632 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:43.223585 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:43.244782 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:43.244731 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" podStartSLOduration=1.354837296 podStartE2EDuration="4.244712479s" totalImagesPullingTime="2.889875183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:43.242789165 +0000 UTC m=+511.764397728" watchObservedRunningTime="2026-07-24 15:55:43.244712479 +0000 UTC m=+511.766321022" Jul 24 15:55:44.610103 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:44.610065 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-478jl"] Jul 24 15:55:46.239750 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:46.239717 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-478jl" event={"ID":"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d","Type":"ContainerStarted","Data":"1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b"} Jul 24 15:55:46.240203 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:46.239755 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-478jl" podUID="aa844b98-b5b9-4ea0-a4bf-bb553edbec2d" containerName="authorino" containerID="cri-o://1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b" gracePeriod=30 Jul 24 15:55:46.252569 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:46.252526 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-478jl" podStartSLOduration=1.8446543260000001 podStartE2EDuration="6.252510249s" totalImagesPullingTime="4.407855923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:46.252154571 +0000 UTC m=+514.773763102" watchObservedRunningTime="2026-07-24 15:55:46.252510249 +0000 UTC m=+514.774118788" Jul 24 15:55:46.487842 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:46.487817 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-478jl" Jul 24 15:55:46.593898 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:46.593858 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bh2p\" (UniqueName: \"kubernetes.io/projected/aa844b98-b5b9-4ea0-a4bf-bb553edbec2d-kube-api-access-4bh2p\") pod \"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d\" (UID: \"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d\") " Jul 24 15:55:46.596245 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:46.596215 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa844b98-b5b9-4ea0-a4bf-bb553edbec2d-kube-api-access-4bh2p" (OuterVolumeSpecName: "kube-api-access-4bh2p") pod "aa844b98-b5b9-4ea0-a4bf-bb553edbec2d" (UID: "aa844b98-b5b9-4ea0-a4bf-bb553edbec2d"). InnerVolumeSpecName "kube-api-access-4bh2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:55:46.694663 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:46.694632 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bh2p\" (UniqueName: \"kubernetes.io/projected/aa844b98-b5b9-4ea0-a4bf-bb553edbec2d-kube-api-access-4bh2p\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:55:47.245141 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.245112 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa844b98-b5b9-4ea0-a4bf-bb553edbec2d" containerID="1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b" exitCode=0 Jul 24 15:55:47.245543 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.245163 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-478jl" Jul 24 15:55:47.245543 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.245162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-478jl" event={"ID":"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d","Type":"ContainerDied","Data":"1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b"} Jul 24 15:55:47.245543 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.245210 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-478jl" event={"ID":"aa844b98-b5b9-4ea0-a4bf-bb553edbec2d","Type":"ContainerDied","Data":"f1ba635e99f88904137ecd03fa4d4460ebb9bbc7b81120498348183ebc57cdc7"} Jul 24 15:55:47.245543 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.245231 2580 scope.go:117] "RemoveContainer" containerID="1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b" Jul 24 15:55:47.254089 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.254070 2580 scope.go:117] "RemoveContainer" containerID="1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b" Jul 24 15:55:47.254381 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:55:47.254364 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b\": container with ID starting with 1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b not found: ID does not exist" containerID="1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b" Jul 24 15:55:47.254421 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.254391 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b"} err="failed to get container status \"1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b\": rpc error: code = NotFound desc = could not find container \"1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b\": container with ID starting with 1a88a14c18af74365f09fe369f7b6e25e008024ff14f2d5cfe474dddb56a023b not found: ID does not exist" Jul 24 15:55:47.265198 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.265173 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-478jl"] Jul 24 15:55:47.269271 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:47.269248 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-478jl"] Jul 24 15:55:48.010204 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:48.010171 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa844b98-b5b9-4ea0-a4bf-bb553edbec2d" path="/var/lib/kubelet/pods/aa844b98-b5b9-4ea0-a4bf-bb553edbec2d/volumes" Jul 24 15:55:54.229957 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:54.229904 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:54.895156 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:54.895121 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8m7g8"] Jul 24 15:55:54.895352 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:54.895310 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" podUID="72536052-6452-4d62-8d3e-2f6ad68f6fe4" containerName="limitador" containerID="cri-o://72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a" gracePeriod=30 Jul 24 15:55:55.450027 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:55.450001 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:55.575954 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:55.575903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfdnw\" (UniqueName: \"kubernetes.io/projected/72536052-6452-4d62-8d3e-2f6ad68f6fe4-kube-api-access-cfdnw\") pod \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " Jul 24 15:55:55.576116 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:55.576057 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72536052-6452-4d62-8d3e-2f6ad68f6fe4-config-file\") pod \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\" (UID: \"72536052-6452-4d62-8d3e-2f6ad68f6fe4\") " Jul 24 15:55:55.576386 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:55.576364 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72536052-6452-4d62-8d3e-2f6ad68f6fe4-config-file" (OuterVolumeSpecName: "config-file") pod "72536052-6452-4d62-8d3e-2f6ad68f6fe4" (UID: "72536052-6452-4d62-8d3e-2f6ad68f6fe4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:55:55.578064 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:55.578039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72536052-6452-4d62-8d3e-2f6ad68f6fe4-kube-api-access-cfdnw" (OuterVolumeSpecName: "kube-api-access-cfdnw") pod "72536052-6452-4d62-8d3e-2f6ad68f6fe4" (UID: "72536052-6452-4d62-8d3e-2f6ad68f6fe4"). InnerVolumeSpecName "kube-api-access-cfdnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:55:55.677386 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:55.677352 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72536052-6452-4d62-8d3e-2f6ad68f6fe4-config-file\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:55:55.677386 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:55.677384 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdnw\" (UniqueName: \"kubernetes.io/projected/72536052-6452-4d62-8d3e-2f6ad68f6fe4-kube-api-access-cfdnw\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:55:56.291417 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.291337 2580 generic.go:356] "Generic (PLEG): container finished" podID="72536052-6452-4d62-8d3e-2f6ad68f6fe4" containerID="72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a" exitCode=0 Jul 24 15:55:56.291417 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.291387 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" Jul 24 15:55:56.291584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.291424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" event={"ID":"72536052-6452-4d62-8d3e-2f6ad68f6fe4","Type":"ContainerDied","Data":"72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a"} Jul 24 15:55:56.291584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.291482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8m7g8" event={"ID":"72536052-6452-4d62-8d3e-2f6ad68f6fe4","Type":"ContainerDied","Data":"4ea59a0bb369e231faaeb278d5339c5b56fb3731e7cd281e6636df6243aa3444"} Jul 24 15:55:56.291584 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.291504 2580 scope.go:117] "RemoveContainer" containerID="72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a" Jul 24 15:55:56.299800 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.299782 2580 scope.go:117] "RemoveContainer" containerID="72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a" Jul 24 15:55:56.300049 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:55:56.300029 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a\": container with ID starting with 72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a not found: ID does not exist" containerID="72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a" Jul 24 15:55:56.300106 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.300057 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a"} err="failed to get container status \"72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a\": rpc error: code = NotFound desc = could not find container \"72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a\": container with ID starting with 72ba5474c90882bd15059fc5bc4152cbcaaaa7b72305a322b571dcb945690d1a not found: ID does not exist" Jul 24 15:55:56.305562 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.305538 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8m7g8"] Jul 24 15:55:56.309410 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:56.309389 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8m7g8"] Jul 24 15:55:58.011160 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:55:58.011128 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72536052-6452-4d62-8d3e-2f6ad68f6fe4" path="/var/lib/kubelet/pods/72536052-6452-4d62-8d3e-2f6ad68f6fe4/volumes" Jul 24 15:56:01.165437 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.165397 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lzq88"] Jul 24 15:56:01.165824 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.165794 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa844b98-b5b9-4ea0-a4bf-bb553edbec2d" containerName="authorino" Jul 24 15:56:01.165824 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.165807 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa844b98-b5b9-4ea0-a4bf-bb553edbec2d" containerName="authorino" Jul 24 15:56:01.165824 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.165817 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72536052-6452-4d62-8d3e-2f6ad68f6fe4" containerName="limitador" Jul 24 15:56:01.165824 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.165822 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72536052-6452-4d62-8d3e-2f6ad68f6fe4" containerName="limitador" Jul 24 15:56:01.165975 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.165885 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa844b98-b5b9-4ea0-a4bf-bb553edbec2d" containerName="authorino" Jul 24 15:56:01.165975 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.165892 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72536052-6452-4d62-8d3e-2f6ad68f6fe4" containerName="limitador" Jul 24 15:56:01.169593 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.169576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.171846 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.171827 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:56:01.171995 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.171881 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9n88t\"" Jul 24 15:56:01.171995 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.171824 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 15:56:01.172115 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.172004 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 15:56:01.176271 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.176248 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lzq88"] Jul 24 15:56:01.324640 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.324609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3879ebc7-7777-49ca-beff-6224945fa871-data\") pod \"postgres-69fb4594cf-lzq88\" (UID: \"3879ebc7-7777-49ca-beff-6224945fa871\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.324811 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.324653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2n2x\" (UniqueName: \"kubernetes.io/projected/3879ebc7-7777-49ca-beff-6224945fa871-kube-api-access-j2n2x\") pod \"postgres-69fb4594cf-lzq88\" (UID: \"3879ebc7-7777-49ca-beff-6224945fa871\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.425695 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.425613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3879ebc7-7777-49ca-beff-6224945fa871-data\") pod \"postgres-69fb4594cf-lzq88\" (UID: \"3879ebc7-7777-49ca-beff-6224945fa871\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.425695 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.425653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2n2x\" (UniqueName: \"kubernetes.io/projected/3879ebc7-7777-49ca-beff-6224945fa871-kube-api-access-j2n2x\") pod \"postgres-69fb4594cf-lzq88\" (UID: \"3879ebc7-7777-49ca-beff-6224945fa871\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.426002 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.425982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3879ebc7-7777-49ca-beff-6224945fa871-data\") pod \"postgres-69fb4594cf-lzq88\" (UID: \"3879ebc7-7777-49ca-beff-6224945fa871\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.436362 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.436334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2n2x\" (UniqueName: \"kubernetes.io/projected/3879ebc7-7777-49ca-beff-6224945fa871-kube-api-access-j2n2x\") pod \"postgres-69fb4594cf-lzq88\" (UID: \"3879ebc7-7777-49ca-beff-6224945fa871\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.481909 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.481887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:01.620199 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:01.620154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lzq88"] Jul 24 15:56:01.622622 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:56:01.622599 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3879ebc7_7777_49ca_beff_6224945fa871.slice/crio-b182f808432ef7a9aae2b54ce6e431e6009d178c43ca0c2808121d429e78a863 WatchSource:0}: Error finding container b182f808432ef7a9aae2b54ce6e431e6009d178c43ca0c2808121d429e78a863: Status 404 returned error can't find the container with id b182f808432ef7a9aae2b54ce6e431e6009d178c43ca0c2808121d429e78a863 Jul 24 15:56:02.319420 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:02.319385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" event={"ID":"3879ebc7-7777-49ca-beff-6224945fa871","Type":"ContainerStarted","Data":"b182f808432ef7a9aae2b54ce6e431e6009d178c43ca0c2808121d429e78a863"} Jul 24 15:56:07.348315 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:07.348273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" event={"ID":"3879ebc7-7777-49ca-beff-6224945fa871","Type":"ContainerStarted","Data":"d32f518574559ee9614f4822f0d22b5467d9958052caba2e062cf2091271a264"} Jul 24 15:56:07.348755 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:07.348332 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:07.364314 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:07.364264 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" podStartSLOduration=1.427591439 podStartE2EDuration="6.364248436s" totalImagesPullingTime="4.936656997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:07.362464489 +0000 UTC m=+535.884073026" watchObservedRunningTime="2026-07-24 15:56:07.364248436 +0000 UTC m=+535.885856975" Jul 24 15:56:13.382255 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:13.382223 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lzq88" Jul 24 15:56:13.914603 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:13.914571 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wl5qb"] Jul 24 15:56:13.921810 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:13.921785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wl5qb"] Jul 24 15:56:13.921921 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:13.921889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" Jul 24 15:56:13.923802 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:13.923780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kwhg4\"" Jul 24 15:56:13.927687 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:13.927664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzvk6\" (UniqueName: \"kubernetes.io/projected/6ed21e61-d200-4f6f-99ff-e935e26c85bb-kube-api-access-gzvk6\") pod \"authorino-5d4b67c446-wl5qb\" (UID: \"6ed21e61-d200-4f6f-99ff-e935e26c85bb\") " pod="kuadrant-system/authorino-5d4b67c446-wl5qb" Jul 24 15:56:14.028650 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.028618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzvk6\" (UniqueName: \"kubernetes.io/projected/6ed21e61-d200-4f6f-99ff-e935e26c85bb-kube-api-access-gzvk6\") pod \"authorino-5d4b67c446-wl5qb\" (UID: \"6ed21e61-d200-4f6f-99ff-e935e26c85bb\") " pod="kuadrant-system/authorino-5d4b67c446-wl5qb" Jul 24 15:56:14.036631 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.036601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzvk6\" (UniqueName: \"kubernetes.io/projected/6ed21e61-d200-4f6f-99ff-e935e26c85bb-kube-api-access-gzvk6\") pod \"authorino-5d4b67c446-wl5qb\" (UID: \"6ed21e61-d200-4f6f-99ff-e935e26c85bb\") " pod="kuadrant-system/authorino-5d4b67c446-wl5qb" Jul 24 15:56:14.146356 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.146323 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wl5qb"] Jul 24 15:56:14.146606 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.146591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" Jul 24 15:56:14.173427 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.173363 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f654bb864-nglsl"] Jul 24 15:56:14.178137 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.178119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f654bb864-nglsl" Jul 24 15:56:14.182435 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.182410 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f654bb864-nglsl"] Jul 24 15:56:14.231023 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.230988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvh7s\" (UniqueName: \"kubernetes.io/projected/ea25f245-76da-4bfc-87e0-eb1e052212b6-kube-api-access-hvh7s\") pod \"authorino-7f654bb864-nglsl\" (UID: \"ea25f245-76da-4bfc-87e0-eb1e052212b6\") " pod="kuadrant-system/authorino-7f654bb864-nglsl" Jul 24 15:56:14.282474 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.282450 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wl5qb"] Jul 24 15:56:14.285417 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:56:14.285139 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ed21e61_d200_4f6f_99ff_e935e26c85bb.slice/crio-4edc76d1af31602022e10c41b7edc0bce2806e423f4937c616299162f1c497d0 WatchSource:0}: Error finding container 4edc76d1af31602022e10c41b7edc0bce2806e423f4937c616299162f1c497d0: Status 404 returned error can't find the container with id 4edc76d1af31602022e10c41b7edc0bce2806e423f4937c616299162f1c497d0 Jul 24 15:56:14.287152 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.287040 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f654bb864-nglsl"] Jul 24 15:56:14.287361 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:56:14.287330 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hvh7s], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7f654bb864-nglsl" podUID="ea25f245-76da-4bfc-87e0-eb1e052212b6" Jul 24 15:56:14.332440 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.332410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvh7s\" (UniqueName: \"kubernetes.io/projected/ea25f245-76da-4bfc-87e0-eb1e052212b6-kube-api-access-hvh7s\") pod \"authorino-7f654bb864-nglsl\" (UID: \"ea25f245-76da-4bfc-87e0-eb1e052212b6\") " pod="kuadrant-system/authorino-7f654bb864-nglsl" Jul 24 15:56:14.339882 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.339861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvh7s\" (UniqueName: \"kubernetes.io/projected/ea25f245-76da-4bfc-87e0-eb1e052212b6-kube-api-access-hvh7s\") pod \"authorino-7f654bb864-nglsl\" (UID: \"ea25f245-76da-4bfc-87e0-eb1e052212b6\") " pod="kuadrant-system/authorino-7f654bb864-nglsl" Jul 24 15:56:14.382025 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.382004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f654bb864-nglsl" Jul 24 15:56:14.382156 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.382056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" event={"ID":"6ed21e61-d200-4f6f-99ff-e935e26c85bb","Type":"ContainerStarted","Data":"4edc76d1af31602022e10c41b7edc0bce2806e423f4937c616299162f1c497d0"} Jul 24 15:56:14.389123 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.389104 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f654bb864-nglsl" Jul 24 15:56:14.433095 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.433034 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvh7s\" (UniqueName: \"kubernetes.io/projected/ea25f245-76da-4bfc-87e0-eb1e052212b6-kube-api-access-hvh7s\") pod \"ea25f245-76da-4bfc-87e0-eb1e052212b6\" (UID: \"ea25f245-76da-4bfc-87e0-eb1e052212b6\") " Jul 24 15:56:14.435107 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.435088 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea25f245-76da-4bfc-87e0-eb1e052212b6-kube-api-access-hvh7s" (OuterVolumeSpecName: "kube-api-access-hvh7s") pod "ea25f245-76da-4bfc-87e0-eb1e052212b6" (UID: "ea25f245-76da-4bfc-87e0-eb1e052212b6"). InnerVolumeSpecName "kube-api-access-hvh7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:56:14.533661 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:14.533617 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvh7s\" (UniqueName: \"kubernetes.io/projected/ea25f245-76da-4bfc-87e0-eb1e052212b6-kube-api-access-hvh7s\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:56:15.388073 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.387989 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f654bb864-nglsl" Jul 24 15:56:15.388073 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.388043 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" event={"ID":"6ed21e61-d200-4f6f-99ff-e935e26c85bb","Type":"ContainerStarted","Data":"fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e"} Jul 24 15:56:15.388335 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.388292 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" podUID="6ed21e61-d200-4f6f-99ff-e935e26c85bb" containerName="authorino" containerID="cri-o://fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e" gracePeriod=30 Jul 24 15:56:15.405207 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.405170 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" podStartSLOduration=1.907872697 podStartE2EDuration="2.405155107s" totalImagesPullingTime="497.28241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:15.403713112 +0000 UTC m=+543.925321651" watchObservedRunningTime="2026-07-24 15:56:15.405155107 +0000 UTC m=+543.926763685" Jul 24 15:56:15.423144 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.423123 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f654bb864-nglsl"] Jul 24 15:56:15.428876 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.428853 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f654bb864-nglsl"] Jul 24 15:56:15.640406 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.640352 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" Jul 24 15:56:15.744079 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.744050 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzvk6\" (UniqueName: \"kubernetes.io/projected/6ed21e61-d200-4f6f-99ff-e935e26c85bb-kube-api-access-gzvk6\") pod \"6ed21e61-d200-4f6f-99ff-e935e26c85bb\" (UID: \"6ed21e61-d200-4f6f-99ff-e935e26c85bb\") " Jul 24 15:56:15.746402 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.746376 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed21e61-d200-4f6f-99ff-e935e26c85bb-kube-api-access-gzvk6" (OuterVolumeSpecName: "kube-api-access-gzvk6") pod "6ed21e61-d200-4f6f-99ff-e935e26c85bb" (UID: "6ed21e61-d200-4f6f-99ff-e935e26c85bb"). InnerVolumeSpecName "kube-api-access-gzvk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:56:15.844618 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:15.844589 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzvk6\" (UniqueName: \"kubernetes.io/projected/6ed21e61-d200-4f6f-99ff-e935e26c85bb-kube-api-access-gzvk6\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:56:16.010112 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.010080 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea25f245-76da-4bfc-87e0-eb1e052212b6" path="/var/lib/kubelet/pods/ea25f245-76da-4bfc-87e0-eb1e052212b6/volumes" Jul 24 15:56:16.392966 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.392872 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ed21e61-d200-4f6f-99ff-e935e26c85bb" containerID="fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e" exitCode=0 Jul 24 15:56:16.392966 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.392909 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" Jul 24 15:56:16.392966 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.392953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" event={"ID":"6ed21e61-d200-4f6f-99ff-e935e26c85bb","Type":"ContainerDied","Data":"fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e"} Jul 24 15:56:16.393217 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.392984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wl5qb" event={"ID":"6ed21e61-d200-4f6f-99ff-e935e26c85bb","Type":"ContainerDied","Data":"4edc76d1af31602022e10c41b7edc0bce2806e423f4937c616299162f1c497d0"} Jul 24 15:56:16.393217 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.393003 2580 scope.go:117] "RemoveContainer" containerID="fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e" Jul 24 15:56:16.401411 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.401400 2580 scope.go:117] "RemoveContainer" containerID="fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e" Jul 24 15:56:16.401663 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:56:16.401645 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e\": container with ID starting with fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e not found: ID does not exist" containerID="fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e" Jul 24 15:56:16.401720 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.401672 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e"} err="failed to get container status \"fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e\": rpc error: code = NotFound desc = could not find container \"fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e\": container with ID starting with fdab5953d2b001e5af0629a98d74d2ef673649cc5baa0ae669c64212579e918e not found: ID does not exist" Jul 24 15:56:16.414231 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.414209 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wl5qb"] Jul 24 15:56:16.417790 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:16.417767 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wl5qb"] Jul 24 15:56:18.010250 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:18.010221 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed21e61-d200-4f6f-99ff-e935e26c85bb" path="/var/lib/kubelet/pods/6ed21e61-d200-4f6f-99ff-e935e26c85bb/volumes" Jul 24 15:56:20.492144 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.492111 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65bb8758ff-zp77f"] Jul 24 15:56:20.492492 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.492477 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed21e61-d200-4f6f-99ff-e935e26c85bb" containerName="authorino" Jul 24 15:56:20.492492 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.492493 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed21e61-d200-4f6f-99ff-e935e26c85bb" containerName="authorino" Jul 24 15:56:20.492562 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.492556 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed21e61-d200-4f6f-99ff-e935e26c85bb" containerName="authorino" Jul 24 15:56:20.498152 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.498122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:20.500538 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.500507 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bzlgk\"" Jul 24 15:56:20.501027 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.501003 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:56:20.501354 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.501076 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:56:20.506594 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.506440 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb8758ff-zp77f"] Jul 24 15:56:20.581168 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.581132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert\") pod \"maas-controller-65bb8758ff-zp77f\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:20.581358 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.581190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75p4f\" (UniqueName: \"kubernetes.io/projected/41ed62cf-00f7-4131-a59d-dbf98ac51078-kube-api-access-75p4f\") pod \"maas-controller-65bb8758ff-zp77f\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:20.682305 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.682260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75p4f\" (UniqueName: \"kubernetes.io/projected/41ed62cf-00f7-4131-a59d-dbf98ac51078-kube-api-access-75p4f\") pod \"maas-controller-65bb8758ff-zp77f\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:20.682476 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.682379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert\") pod \"maas-controller-65bb8758ff-zp77f\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:20.682538 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:56:20.682517 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 15:56:20.682603 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:56:20.682594 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert podName:41ed62cf-00f7-4131-a59d-dbf98ac51078 nodeName:}" failed. No retries permitted until 2026-07-24 15:56:21.182577582 +0000 UTC m=+549.704186104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert") pod "maas-controller-65bb8758ff-zp77f" (UID: "41ed62cf-00f7-4131-a59d-dbf98ac51078") : secret "maas-controller-webhook-cert" not found Jul 24 15:56:20.688838 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.688813 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc959d7c4-gkw57"] Jul 24 15:56:20.691057 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.691043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:20.700005 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.699985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75p4f\" (UniqueName: \"kubernetes.io/projected/41ed62cf-00f7-4131-a59d-dbf98ac51078-kube-api-access-75p4f\") pod \"maas-controller-65bb8758ff-zp77f\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:20.775445 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.775413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-gkw57"] Jul 24 15:56:20.783576 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.783546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert\") pod \"maas-controller-cc959d7c4-gkw57\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:20.783719 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.783589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmxg\" (UniqueName: \"kubernetes.io/projected/f9c73952-f653-4f63-bd01-0b9665618755-kube-api-access-mkmxg\") pod \"maas-controller-cc959d7c4-gkw57\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:20.884235 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.884185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert\") pod \"maas-controller-cc959d7c4-gkw57\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:20.884422 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.884259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmxg\" (UniqueName: \"kubernetes.io/projected/f9c73952-f653-4f63-bd01-0b9665618755-kube-api-access-mkmxg\") pod \"maas-controller-cc959d7c4-gkw57\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:20.884422 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:56:20.884354 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 15:56:20.884496 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:56:20.884423 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert podName:f9c73952-f653-4f63-bd01-0b9665618755 nodeName:}" failed. No retries permitted until 2026-07-24 15:56:21.384409594 +0000 UTC m=+549.906018111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert") pod "maas-controller-cc959d7c4-gkw57" (UID: "f9c73952-f653-4f63-bd01-0b9665618755") : secret "maas-controller-webhook-cert" not found Jul 24 15:56:20.894680 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:20.894658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmxg\" (UniqueName: \"kubernetes.io/projected/f9c73952-f653-4f63-bd01-0b9665618755-kube-api-access-mkmxg\") pod \"maas-controller-cc959d7c4-gkw57\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:21.186738 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.186650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert\") pod \"maas-controller-65bb8758ff-zp77f\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:21.189219 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.189199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert\") pod \"maas-controller-65bb8758ff-zp77f\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:21.388509 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.388454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert\") pod \"maas-controller-cc959d7c4-gkw57\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:21.391059 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.391039 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert\") pod \"maas-controller-cc959d7c4-gkw57\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:21.411998 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.411963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:21.542698 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.542673 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb8758ff-zp77f"] Jul 24 15:56:21.545182 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:56:21.545160 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ed62cf_00f7_4131_a59d_dbf98ac51078.slice/crio-a8e724aae5cb8951205bc4c07258ab0f914edca90c46c78acde64b3a529cd68d WatchSource:0}: Error finding container a8e724aae5cb8951205bc4c07258ab0f914edca90c46c78acde64b3a529cd68d: Status 404 returned error can't find the container with id a8e724aae5cb8951205bc4c07258ab0f914edca90c46c78acde64b3a529cd68d Jul 24 15:56:21.601269 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.601243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:21.729215 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:21.729151 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-gkw57"] Jul 24 15:56:21.732698 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:56:21.732675 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c73952_f653_4f63_bd01_0b9665618755.slice/crio-169210abe70ec66cf6741702384171a76cda6f19e586215270962c6c0ee046e4 WatchSource:0}: Error finding container 169210abe70ec66cf6741702384171a76cda6f19e586215270962c6c0ee046e4: Status 404 returned error can't find the container with id 169210abe70ec66cf6741702384171a76cda6f19e586215270962c6c0ee046e4 Jul 24 15:56:22.427562 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:22.427526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb8758ff-zp77f" event={"ID":"41ed62cf-00f7-4131-a59d-dbf98ac51078","Type":"ContainerStarted","Data":"a8e724aae5cb8951205bc4c07258ab0f914edca90c46c78acde64b3a529cd68d"} Jul 24 15:56:22.430792 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:22.430763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-gkw57" event={"ID":"f9c73952-f653-4f63-bd01-0b9665618755","Type":"ContainerStarted","Data":"169210abe70ec66cf6741702384171a76cda6f19e586215270962c6c0ee046e4"} Jul 24 15:56:24.443508 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:24.443472 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb8758ff-zp77f" event={"ID":"41ed62cf-00f7-4131-a59d-dbf98ac51078","Type":"ContainerStarted","Data":"a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f"} Jul 24 15:56:24.443973 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:24.443589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:24.445264 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:24.445201 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-gkw57" event={"ID":"f9c73952-f653-4f63-bd01-0b9665618755","Type":"ContainerStarted","Data":"0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0"} Jul 24 15:56:24.445264 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:24.445254 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:24.463197 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:24.463151 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65bb8758ff-zp77f" podStartSLOduration=1.803712316 podStartE2EDuration="4.463135528s" totalImagesPullingTime="2.659423212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:24.461507889 +0000 UTC m=+552.983116429" watchObservedRunningTime="2026-07-24 15:56:24.463135528 +0000 UTC m=+552.984744090" Jul 24 15:56:24.481843 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:24.481792 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc959d7c4-gkw57" podStartSLOduration=2.016294943 podStartE2EDuration="4.481777388s" totalImagesPullingTime="2.465482445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:24.481036988 +0000 UTC m=+553.002645538" watchObservedRunningTime="2026-07-24 15:56:24.481777388 +0000 UTC m=+553.003385924" Jul 24 15:56:35.451016 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.450985 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 15:56:35.451476 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.451068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:35.509496 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.509468 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65bb8758ff-zp77f"] Jul 24 15:56:35.509660 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.509623 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65bb8758ff-zp77f" podUID="41ed62cf-00f7-4131-a59d-dbf98ac51078" containerName="manager" containerID="cri-o://a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f" gracePeriod=30 Jul 24 15:56:35.745148 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.745128 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:35.819035 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.819002 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75p4f\" (UniqueName: \"kubernetes.io/projected/41ed62cf-00f7-4131-a59d-dbf98ac51078-kube-api-access-75p4f\") pod \"41ed62cf-00f7-4131-a59d-dbf98ac51078\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " Jul 24 15:56:35.819182 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.819104 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert\") pod \"41ed62cf-00f7-4131-a59d-dbf98ac51078\" (UID: \"41ed62cf-00f7-4131-a59d-dbf98ac51078\") " Jul 24 15:56:35.821381 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.821352 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ed62cf-00f7-4131-a59d-dbf98ac51078-kube-api-access-75p4f" (OuterVolumeSpecName: "kube-api-access-75p4f") pod "41ed62cf-00f7-4131-a59d-dbf98ac51078" (UID: "41ed62cf-00f7-4131-a59d-dbf98ac51078"). InnerVolumeSpecName "kube-api-access-75p4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:56:35.821486 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.821416 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "41ed62cf-00f7-4131-a59d-dbf98ac51078" (UID: "41ed62cf-00f7-4131-a59d-dbf98ac51078"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:56:35.920243 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.920217 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41ed62cf-00f7-4131-a59d-dbf98ac51078-webhook-cert\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:56:35.920243 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:35.920240 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75p4f\" (UniqueName: \"kubernetes.io/projected/41ed62cf-00f7-4131-a59d-dbf98ac51078-kube-api-access-75p4f\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 15:56:36.503727 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.503637 2580 generic.go:356] "Generic (PLEG): container finished" podID="41ed62cf-00f7-4131-a59d-dbf98ac51078" containerID="a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f" exitCode=0 Jul 24 15:56:36.503727 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.503680 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb8758ff-zp77f" Jul 24 15:56:36.504218 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.503724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb8758ff-zp77f" event={"ID":"41ed62cf-00f7-4131-a59d-dbf98ac51078","Type":"ContainerDied","Data":"a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f"} Jul 24 15:56:36.504218 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.503784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb8758ff-zp77f" event={"ID":"41ed62cf-00f7-4131-a59d-dbf98ac51078","Type":"ContainerDied","Data":"a8e724aae5cb8951205bc4c07258ab0f914edca90c46c78acde64b3a529cd68d"} Jul 24 15:56:36.504218 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.503807 2580 scope.go:117] "RemoveContainer" containerID="a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f" Jul 24 15:56:36.512677 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.512532 2580 scope.go:117] "RemoveContainer" containerID="a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f" Jul 24 15:56:36.512808 ip-10-0-140-155 kubenswrapper[2580]: E0724 15:56:36.512788 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f\": container with ID starting with a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f not found: ID does not exist" containerID="a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f" Jul 24 15:56:36.512860 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.512816 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f"} err="failed to get container status \"a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f\": rpc error: code = NotFound desc = could not find container \"a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f\": container with ID starting with a77df79fc1837388c343cee9c98d1e835de0c41bc04a96fdd67bfb661b83650f not found: ID does not exist" Jul 24 15:56:36.519846 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.519823 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65bb8758ff-zp77f"] Jul 24 15:56:36.523426 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:36.523408 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65bb8758ff-zp77f"] Jul 24 15:56:38.010159 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:38.010121 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ed62cf-00f7-4131-a59d-dbf98ac51078" path="/var/lib/kubelet/pods/41ed62cf-00f7-4131-a59d-dbf98ac51078/volumes" Jul 24 15:56:56.917043 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.917009 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf"] Jul 24 15:56:56.917456 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.917369 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ed62cf-00f7-4131-a59d-dbf98ac51078" containerName="manager" Jul 24 15:56:56.917456 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.917382 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ed62cf-00f7-4131-a59d-dbf98ac51078" containerName="manager" Jul 24 15:56:56.917456 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.917455 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ed62cf-00f7-4131-a59d-dbf98ac51078" containerName="manager" Jul 24 15:56:56.959863 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.959830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf"] Jul 24 15:56:56.960042 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.959977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:56.962379 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.962362 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 15:56:56.962484 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:56.962472 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6xp64\"" Jul 24 15:56:57.108308 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.108278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfd6a9a-9c84-42c0-87d8-0364266c7403-openshift-service-ca\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.108308 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.108314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdf8g\" (UniqueName: \"kubernetes.io/projected/2dfd6a9a-9c84-42c0-87d8-0364266c7403-kube-api-access-bdf8g\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.108523 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.108377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfd6a9a-9c84-42c0-87d8-0364266c7403-maas-api-tls\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.209777 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.209692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfd6a9a-9c84-42c0-87d8-0364266c7403-openshift-service-ca\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.209777 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.209727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdf8g\" (UniqueName: \"kubernetes.io/projected/2dfd6a9a-9c84-42c0-87d8-0364266c7403-kube-api-access-bdf8g\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.210033 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.209867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfd6a9a-9c84-42c0-87d8-0364266c7403-maas-api-tls\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.210372 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.210352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfd6a9a-9c84-42c0-87d8-0364266c7403-openshift-service-ca\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.212360 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.212338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfd6a9a-9c84-42c0-87d8-0364266c7403-maas-api-tls\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.222564 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.222538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdf8g\" (UniqueName: \"kubernetes.io/projected/2dfd6a9a-9c84-42c0-87d8-0364266c7403-kube-api-access-bdf8g\") pod \"maas-api-6df5894bdc-v2rgf\" (UID: \"2dfd6a9a-9c84-42c0-87d8-0364266c7403\") " pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.270283 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.270249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:56:57.336143 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.334298 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9czsr"] Jul 24 15:56:57.338012 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.337990 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.340113 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.340087 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 15:56:57.340248 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.340169 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lz2zg\"" Jul 24 15:56:57.352181 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.352144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9czsr"] Jul 24 15:56:57.412695 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.412671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf"] Jul 24 15:56:57.415026 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:56:57.414986 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfd6a9a_9c84_42c0_87d8_0364266c7403.slice/crio-0f3484f08ab4000a17b3fbbd569787ed5ad09f1cc9ac41a7b172096e2de9b364 WatchSource:0}: Error finding container 0f3484f08ab4000a17b3fbbd569787ed5ad09f1cc9ac41a7b172096e2de9b364: Status 404 returned error can't find the container with id 0f3484f08ab4000a17b3fbbd569787ed5ad09f1cc9ac41a7b172096e2de9b364 Jul 24 15:56:57.512723 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.512687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba87454-6133-46b3-a7f1-1f2053b04e98-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9czsr\" (UID: \"aba87454-6133-46b3-a7f1-1f2053b04e98\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.512723 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.512726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vksfv\" (UniqueName: \"kubernetes.io/projected/aba87454-6133-46b3-a7f1-1f2053b04e98-kube-api-access-vksfv\") pod \"payload-processing-5cc6cd444b-9czsr\" (UID: \"aba87454-6133-46b3-a7f1-1f2053b04e98\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.599006 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.598970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" event={"ID":"2dfd6a9a-9c84-42c0-87d8-0364266c7403","Type":"ContainerStarted","Data":"0f3484f08ab4000a17b3fbbd569787ed5ad09f1cc9ac41a7b172096e2de9b364"} Jul 24 15:56:57.613419 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.613388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba87454-6133-46b3-a7f1-1f2053b04e98-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9czsr\" (UID: \"aba87454-6133-46b3-a7f1-1f2053b04e98\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.613563 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.613424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vksfv\" (UniqueName: \"kubernetes.io/projected/aba87454-6133-46b3-a7f1-1f2053b04e98-kube-api-access-vksfv\") pod \"payload-processing-5cc6cd444b-9czsr\" (UID: \"aba87454-6133-46b3-a7f1-1f2053b04e98\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.614160 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.614142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba87454-6133-46b3-a7f1-1f2053b04e98-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9czsr\" (UID: \"aba87454-6133-46b3-a7f1-1f2053b04e98\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.623818 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.623795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vksfv\" (UniqueName: \"kubernetes.io/projected/aba87454-6133-46b3-a7f1-1f2053b04e98-kube-api-access-vksfv\") pod \"payload-processing-5cc6cd444b-9czsr\" (UID: \"aba87454-6133-46b3-a7f1-1f2053b04e98\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.650686 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.650655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:56:57.784430 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:57.784408 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9czsr"] Jul 24 15:56:57.786995 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:56:57.786971 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba87454_6133_46b3_a7f1_1f2053b04e98.slice/crio-6c8e5144adfaf09580770f8776bacd550e693dee29b3d4e7571e55a3a7597e44 WatchSource:0}: Error finding container 6c8e5144adfaf09580770f8776bacd550e693dee29b3d4e7571e55a3a7597e44: Status 404 returned error can't find the container with id 6c8e5144adfaf09580770f8776bacd550e693dee29b3d4e7571e55a3a7597e44 Jul 24 15:56:58.607848 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:56:58.607807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" event={"ID":"aba87454-6133-46b3-a7f1-1f2053b04e98","Type":"ContainerStarted","Data":"6c8e5144adfaf09580770f8776bacd550e693dee29b3d4e7571e55a3a7597e44"} Jul 24 15:57:00.619787 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:00.619753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" event={"ID":"2dfd6a9a-9c84-42c0-87d8-0364266c7403","Type":"ContainerStarted","Data":"64659f93677db1bfe9784a0dd64c3bdc76b8b37b4aeebbd8d05ae3193b1fb5ac"} Jul 24 15:57:00.620268 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:00.619868 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:57:00.621365 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:00.621342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" event={"ID":"aba87454-6133-46b3-a7f1-1f2053b04e98","Type":"ContainerStarted","Data":"67c3957e263835782635558e208f1dfb439dd86ca61a65c9366d51cc2ce2c397"} Jul 24 15:57:00.621470 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:00.621393 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:57:00.633534 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:00.633493 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" podStartSLOduration=2.322880776 podStartE2EDuration="4.63347747s" totalImagesPullingTime="2.310596694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:57:00.633161079 +0000 UTC m=+589.154769630" watchObservedRunningTime="2026-07-24 15:57:00.63347747 +0000 UTC m=+589.155086012" Jul 24 15:57:00.648035 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:00.647997 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" podStartSLOduration=1.708479513 podStartE2EDuration="3.647984108s" totalImagesPullingTime="1.939504595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:57:00.64669274 +0000 UTC m=+589.168301279" watchObservedRunningTime="2026-07-24 15:57:00.647984108 +0000 UTC m=+589.169592647" Jul 24 15:57:06.632692 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:06.632666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6df5894bdc-v2rgf" Jul 24 15:57:11.629819 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:11.629792 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9czsr" Jul 24 15:57:11.935167 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:11.935093 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 15:57:11.936952 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:11.936913 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 15:57:31.988528 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:31.988492 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb"] Jul 24 15:57:31.992841 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:31.992824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:31.994794 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:31.994760 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 15:57:31.995482 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:31.995461 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 15:57:31.995482 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:31.995474 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zqxbv\"" Jul 24 15:57:31.995641 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:31.995490 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 15:57:32.001644 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.001621 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb"] Jul 24 15:57:32.105225 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.105195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.105225 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.105229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5v8v\" (UniqueName: \"kubernetes.io/projected/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-kube-api-access-n5v8v\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.105415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.105255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.105415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.105311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.105415 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.105369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.105534 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.105439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.206638 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.206614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.206755 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.206662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.206803 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.206782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.206838 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.206822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5v8v\" (UniqueName: \"kubernetes.io/projected/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-kube-api-access-n5v8v\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.206878 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.206851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.206919 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.206898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.207085 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.207069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.207136 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.207090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.207257 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.207239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.209123 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.209105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.209474 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.209449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.213999 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.213982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5v8v\" (UniqueName: \"kubernetes.io/projected/2c1669bd-c09d-4a14-afda-9c0d835cc0ce-kube-api-access-n5v8v\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpmdb\" (UID: \"2c1669bd-c09d-4a14-afda-9c0d835cc0ce\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.305677 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.305659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:32.439386 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.439364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb"] Jul 24 15:57:32.441714 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:57:32.441689 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1669bd_c09d_4a14_afda_9c0d835cc0ce.slice/crio-5a2d407e591fa3d52791c2011b67f87a58a93826c73754857444e9f31fc697ff WatchSource:0}: Error finding container 5a2d407e591fa3d52791c2011b67f87a58a93826c73754857444e9f31fc697ff: Status 404 returned error can't find the container with id 5a2d407e591fa3d52791c2011b67f87a58a93826c73754857444e9f31fc697ff Jul 24 15:57:32.779299 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:32.779262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" event={"ID":"2c1669bd-c09d-4a14-afda-9c0d835cc0ce","Type":"ContainerStarted","Data":"5a2d407e591fa3d52791c2011b67f87a58a93826c73754857444e9f31fc697ff"} Jul 24 15:57:34.791086 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:34.791052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" event={"ID":"2c1669bd-c09d-4a14-afda-9c0d835cc0ce","Type":"ContainerStarted","Data":"e41d57ef1a0495f67cd4723450970fc0cc4680285669299fe034e43d737af27b"} Jul 24 15:57:34.808255 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:34.808210 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" podStartSLOduration=2.478064695 podStartE2EDuration="3.808196882s" totalImagesPullingTime="1.330132187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:57:34.806146102 +0000 UTC m=+623.327754642" watchObservedRunningTime="2026-07-24 15:57:34.808196882 +0000 UTC m=+623.329805422" Jul 24 15:57:42.306499 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:42.306457 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:42.306873 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:42.306524 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:42.319258 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:42.319235 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:57:42.843441 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:57:42.843408 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpmdb" Jul 24 15:58:09.893925 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.893889 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx"] Jul 24 15:58:09.898623 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.898607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:09.900635 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.900611 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 24 15:58:09.905006 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.904984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx"] Jul 24 15:58:09.948124 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.948092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:09.948300 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.948134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:09.948300 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.948156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:09.948300 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.948184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98cz6\" (UniqueName: \"kubernetes.io/projected/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-kube-api-access-98cz6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:09.948300 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.948262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:09.948507 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:09.948305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.049672 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.049632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.049672 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.049675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.049916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.049694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.049916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.049722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98cz6\" (UniqueName: \"kubernetes.io/projected/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-kube-api-access-98cz6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.049916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.049749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.049916 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.049774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.050128 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.050106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.050202 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.050183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.050255 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.050221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.052210 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.052191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.052458 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.052440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.056676 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.056653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98cz6\" (UniqueName: \"kubernetes.io/projected/4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7-kube-api-access-98cz6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s79jx\" (UID: \"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.209740 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.209660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:10.348918 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.348891 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx"] Jul 24 15:58:10.350863 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:58:10.350827 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de7c0b6_a191_4b7f_ba50_4e1ffc3511e7.slice/crio-6beb25c0c7a14c088ee49f83810d5e078cb0b1df37f3e155e0a278fa486bb645 WatchSource:0}: Error finding container 6beb25c0c7a14c088ee49f83810d5e078cb0b1df37f3e155e0a278fa486bb645: Status 404 returned error can't find the container with id 6beb25c0c7a14c088ee49f83810d5e078cb0b1df37f3e155e0a278fa486bb645 Jul 24 15:58:10.352671 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.352652 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:58:10.966003 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:10.965968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" event={"ID":"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7","Type":"ContainerStarted","Data":"6beb25c0c7a14c088ee49f83810d5e078cb0b1df37f3e155e0a278fa486bb645"} Jul 24 15:58:11.971571 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:11.971495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" event={"ID":"4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7","Type":"ContainerStarted","Data":"80ae7e01f9d53f5566a04468c22b8300224c50601ca1c0a008acc5fa2b01385e"} Jul 24 15:58:11.986882 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:11.986841 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" podStartSLOduration=1.722885038 podStartE2EDuration="2.986826135s" totalImagesPullingTime="1.263941097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:58:11.986116151 +0000 UTC m=+660.507724708" watchObservedRunningTime="2026-07-24 15:58:11.986826135 +0000 UTC m=+660.508434720" Jul 24 15:58:20.210799 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.210768 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:20.211196 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.210822 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:20.222898 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.222877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:20.390304 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.390272 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj"] Jul 24 15:58:20.394015 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.393997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.395858 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.395843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 24 15:58:20.401864 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.401844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj"] Jul 24 15:58:20.541600 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.541569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.541778 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.541612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.541778 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.541630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.541778 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.541692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.541778 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.541752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhhj\" (UniqueName: \"kubernetes.io/projected/a35d3e38-cf86-4e65-926d-4244c0884234-kube-api-access-zvhhj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.542014 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.541826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35d3e38-cf86-4e65-926d-4244c0884234-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.643339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.643339 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.643581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.643581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.643581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhhj\" (UniqueName: \"kubernetes.io/projected/a35d3e38-cf86-4e65-926d-4244c0884234-kube-api-access-zvhhj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.643581 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35d3e38-cf86-4e65-926d-4244c0884234-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.643955 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.644070 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.644070 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.643982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.645986 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.645959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35d3e38-cf86-4e65-926d-4244c0884234-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.646281 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.646260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35d3e38-cf86-4e65-926d-4244c0884234-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.653597 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.653574 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhhj\" (UniqueName: \"kubernetes.io/projected/a35d3e38-cf86-4e65-926d-4244c0884234-kube-api-access-zvhhj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj\" (UID: \"a35d3e38-cf86-4e65-926d-4244c0884234\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.704488 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.704455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:20.839088 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:20.839060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj"] Jul 24 15:58:20.841376 ip-10-0-140-155 kubenswrapper[2580]: W0724 15:58:20.841347 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda35d3e38_cf86_4e65_926d_4244c0884234.slice/crio-6b36038dcdf38afc210c2bae54a1aa12f3696a6a9bce8e619cb9c96a5514102c WatchSource:0}: Error finding container 6b36038dcdf38afc210c2bae54a1aa12f3696a6a9bce8e619cb9c96a5514102c: Status 404 returned error can't find the container with id 6b36038dcdf38afc210c2bae54a1aa12f3696a6a9bce8e619cb9c96a5514102c Jul 24 15:58:21.014111 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:21.014080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" event={"ID":"a35d3e38-cf86-4e65-926d-4244c0884234","Type":"ContainerStarted","Data":"6b36038dcdf38afc210c2bae54a1aa12f3696a6a9bce8e619cb9c96a5514102c"} Jul 24 15:58:21.025679 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:21.025656 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s79jx" Jul 24 15:58:23.031974 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:23.031918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" event={"ID":"a35d3e38-cf86-4e65-926d-4244c0884234","Type":"ContainerStarted","Data":"aab0dbbbc18b9f121121285238b5da02a35649c33a9e770a167c1388dcd74ac5"} Jul 24 15:58:23.049606 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:23.049557 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" podStartSLOduration=1.8997834710000001 podStartE2EDuration="3.049540071s" totalImagesPullingTime="1.1497566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:58:23.047976252 +0000 UTC m=+671.569584793" watchObservedRunningTime="2026-07-24 15:58:23.049540071 +0000 UTC m=+671.571148610" Jul 24 15:58:30.705203 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:30.705155 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:30.705599 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:30.705223 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:30.717922 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:30.717895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 15:58:31.080865 ip-10-0-140-155 kubenswrapper[2580]: I0724 15:58:31.080836 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj" Jul 24 16:00:17.953065 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:17.953032 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-gkw57"] Jul 24 16:00:17.953560 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:17.953270 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cc959d7c4-gkw57" podUID="f9c73952-f653-4f63-bd01-0b9665618755" containerName="manager" containerID="cri-o://0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0" gracePeriod=30 Jul 24 16:00:18.201158 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.201133 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 16:00:18.300759 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.300736 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert\") pod \"f9c73952-f653-4f63-bd01-0b9665618755\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " Jul 24 16:00:18.300904 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.300787 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkmxg\" (UniqueName: \"kubernetes.io/projected/f9c73952-f653-4f63-bd01-0b9665618755-kube-api-access-mkmxg\") pod \"f9c73952-f653-4f63-bd01-0b9665618755\" (UID: \"f9c73952-f653-4f63-bd01-0b9665618755\") " Jul 24 16:00:18.302890 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.302858 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c73952-f653-4f63-bd01-0b9665618755-kube-api-access-mkmxg" (OuterVolumeSpecName: "kube-api-access-mkmxg") pod "f9c73952-f653-4f63-bd01-0b9665618755" (UID: "f9c73952-f653-4f63-bd01-0b9665618755"). InnerVolumeSpecName "kube-api-access-mkmxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:00:18.303012 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.302977 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f9c73952-f653-4f63-bd01-0b9665618755" (UID: "f9c73952-f653-4f63-bd01-0b9665618755"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:00:18.401783 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.401754 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c73952-f653-4f63-bd01-0b9665618755-webhook-cert\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:00:18.401783 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.401780 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkmxg\" (UniqueName: \"kubernetes.io/projected/f9c73952-f653-4f63-bd01-0b9665618755-kube-api-access-mkmxg\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:00:18.574071 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.573996 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9c73952-f653-4f63-bd01-0b9665618755" containerID="0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0" exitCode=0 Jul 24 16:00:18.574071 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.574063 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-gkw57" Jul 24 16:00:18.574232 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.574082 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-gkw57" event={"ID":"f9c73952-f653-4f63-bd01-0b9665618755","Type":"ContainerDied","Data":"0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0"} Jul 24 16:00:18.574232 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.574158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-gkw57" event={"ID":"f9c73952-f653-4f63-bd01-0b9665618755","Type":"ContainerDied","Data":"169210abe70ec66cf6741702384171a76cda6f19e586215270962c6c0ee046e4"} Jul 24 16:00:18.574232 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.574180 2580 scope.go:117] "RemoveContainer" containerID="0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0" Jul 24 16:00:18.583749 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.583731 2580 scope.go:117] "RemoveContainer" containerID="0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0" Jul 24 16:00:18.584034 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:00:18.584016 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0\": container with ID starting with 0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0 not found: ID does not exist" containerID="0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0" Jul 24 16:00:18.584103 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.584041 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0"} err="failed to get container status \"0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0\": rpc error: code = NotFound desc = could not find container \"0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0\": container with ID starting with 0804e7a430c478d1a73d27fa340be8b4fe1123e1d87274215bb17f9c5d21a7d0 not found: ID does not exist" Jul 24 16:00:18.601879 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.601857 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-gkw57"] Jul 24 16:00:18.610665 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:18.610646 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-gkw57"] Jul 24 16:00:20.010195 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:00:20.010165 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c73952-f653-4f63-bd01-0b9665618755" path="/var/lib/kubelet/pods/f9c73952-f653-4f63-bd01-0b9665618755/volumes" Jul 24 16:01:19.411626 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.411595 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc959d7c4-cfzt7"] Jul 24 16:01:19.412086 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.411989 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c73952-f653-4f63-bd01-0b9665618755" containerName="manager" Jul 24 16:01:19.412086 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.412003 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c73952-f653-4f63-bd01-0b9665618755" containerName="manager" Jul 24 16:01:19.412086 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.412065 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c73952-f653-4f63-bd01-0b9665618755" containerName="manager" Jul 24 16:01:19.416414 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.416399 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.418874 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.418851 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 16:01:19.419425 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.419409 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 16:01:19.419514 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.419460 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bzlgk\"" Jul 24 16:01:19.429565 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.429544 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-cfzt7"] Jul 24 16:01:19.454545 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.454519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8633ef8f-8873-4c60-9844-cff9b2a083ef-webhook-cert\") pod \"maas-controller-cc959d7c4-cfzt7\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.454642 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.454551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5zf8\" (UniqueName: \"kubernetes.io/projected/8633ef8f-8873-4c60-9844-cff9b2a083ef-kube-api-access-l5zf8\") pod \"maas-controller-cc959d7c4-cfzt7\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.555150 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.555123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8633ef8f-8873-4c60-9844-cff9b2a083ef-webhook-cert\") pod \"maas-controller-cc959d7c4-cfzt7\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.555150 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.555153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zf8\" (UniqueName: \"kubernetes.io/projected/8633ef8f-8873-4c60-9844-cff9b2a083ef-kube-api-access-l5zf8\") pod \"maas-controller-cc959d7c4-cfzt7\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.557622 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.557601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8633ef8f-8873-4c60-9844-cff9b2a083ef-webhook-cert\") pod \"maas-controller-cc959d7c4-cfzt7\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.564352 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.564327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zf8\" (UniqueName: \"kubernetes.io/projected/8633ef8f-8873-4c60-9844-cff9b2a083ef-kube-api-access-l5zf8\") pod \"maas-controller-cc959d7c4-cfzt7\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.726646 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.726542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:19.859755 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.859730 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-cfzt7"] Jul 24 16:01:19.861906 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:01:19.861877 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8633ef8f_8873_4c60_9844_cff9b2a083ef.slice/crio-79d4a7dc67e83ac938337e555f1eab2b717656ae1ea7a8d09c59dfede67b6088 WatchSource:0}: Error finding container 79d4a7dc67e83ac938337e555f1eab2b717656ae1ea7a8d09c59dfede67b6088: Status 404 returned error can't find the container with id 79d4a7dc67e83ac938337e555f1eab2b717656ae1ea7a8d09c59dfede67b6088 Jul 24 16:01:19.878637 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:19.878609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" event={"ID":"8633ef8f-8873-4c60-9844-cff9b2a083ef","Type":"ContainerStarted","Data":"79d4a7dc67e83ac938337e555f1eab2b717656ae1ea7a8d09c59dfede67b6088"} Jul 24 16:01:20.884373 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:20.884334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" event={"ID":"8633ef8f-8873-4c60-9844-cff9b2a083ef","Type":"ContainerStarted","Data":"e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9"} Jul 24 16:01:20.884757 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:20.884468 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:20.903512 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:20.903468 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" podStartSLOduration=1.42122336 podStartE2EDuration="1.903453773s" totalImagesPullingTime="482.230413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:01:20.902531544 +0000 UTC m=+849.424140084" watchObservedRunningTime="2026-07-24 16:01:20.903453773 +0000 UTC m=+849.425062339" Jul 24 16:01:31.891365 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:31.891336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:36.549645 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.549611 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-cfzt7"] Jul 24 16:01:36.550061 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.549804 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" podUID="8633ef8f-8873-4c60-9844-cff9b2a083ef" containerName="manager" containerID="cri-o://e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9" gracePeriod=30 Jul 24 16:01:36.793161 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.793135 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:36.895953 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.895853 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8633ef8f-8873-4c60-9844-cff9b2a083ef-webhook-cert\") pod \"8633ef8f-8873-4c60-9844-cff9b2a083ef\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " Jul 24 16:01:36.895953 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.895898 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5zf8\" (UniqueName: \"kubernetes.io/projected/8633ef8f-8873-4c60-9844-cff9b2a083ef-kube-api-access-l5zf8\") pod \"8633ef8f-8873-4c60-9844-cff9b2a083ef\" (UID: \"8633ef8f-8873-4c60-9844-cff9b2a083ef\") " Jul 24 16:01:36.897949 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.897909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8633ef8f-8873-4c60-9844-cff9b2a083ef-kube-api-access-l5zf8" (OuterVolumeSpecName: "kube-api-access-l5zf8") pod "8633ef8f-8873-4c60-9844-cff9b2a083ef" (UID: "8633ef8f-8873-4c60-9844-cff9b2a083ef"). InnerVolumeSpecName "kube-api-access-l5zf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:01:36.898059 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.897951 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8633ef8f-8873-4c60-9844-cff9b2a083ef-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8633ef8f-8873-4c60-9844-cff9b2a083ef" (UID: "8633ef8f-8873-4c60-9844-cff9b2a083ef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:01:36.957604 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.957578 2580 generic.go:356] "Generic (PLEG): container finished" podID="8633ef8f-8873-4c60-9844-cff9b2a083ef" containerID="e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9" exitCode=0 Jul 24 16:01:36.957728 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.957620 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" Jul 24 16:01:36.957728 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.957661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" event={"ID":"8633ef8f-8873-4c60-9844-cff9b2a083ef","Type":"ContainerDied","Data":"e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9"} Jul 24 16:01:36.957728 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.957698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-cfzt7" event={"ID":"8633ef8f-8873-4c60-9844-cff9b2a083ef","Type":"ContainerDied","Data":"79d4a7dc67e83ac938337e555f1eab2b717656ae1ea7a8d09c59dfede67b6088"} Jul 24 16:01:36.957728 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.957720 2580 scope.go:117] "RemoveContainer" containerID="e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9" Jul 24 16:01:36.966786 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.966768 2580 scope.go:117] "RemoveContainer" containerID="e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9" Jul 24 16:01:36.967065 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:01:36.967045 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9\": container with ID starting with e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9 not found: ID does not exist" containerID="e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9" Jul 24 16:01:36.967124 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.967075 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9"} err="failed to get container status \"e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9\": rpc error: code = NotFound desc = could not find container \"e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9\": container with ID starting with e8d968754715665bce3ba9a2f7942704debff762c79cc8fcfb57cb68fc79b5c9 not found: ID does not exist" Jul 24 16:01:36.988321 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.988291 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-cfzt7"] Jul 24 16:01:36.991584 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.991565 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-cfzt7"] Jul 24 16:01:36.997084 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.997060 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8633ef8f-8873-4c60-9844-cff9b2a083ef-webhook-cert\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:01:36.997084 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:36.997080 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5zf8\" (UniqueName: \"kubernetes.io/projected/8633ef8f-8873-4c60-9844-cff9b2a083ef-kube-api-access-l5zf8\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:01:37.759137 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.759102 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cc959d7c4-vk82w"] Jul 24 16:01:37.759538 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.759523 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8633ef8f-8873-4c60-9844-cff9b2a083ef" containerName="manager" Jul 24 16:01:37.759581 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.759540 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8633ef8f-8873-4c60-9844-cff9b2a083ef" containerName="manager" Jul 24 16:01:37.759617 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.759605 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8633ef8f-8873-4c60-9844-cff9b2a083ef" containerName="manager" Jul 24 16:01:37.764013 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.763996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:37.766718 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.766694 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 16:01:37.766835 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.766697 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 16:01:37.766835 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.766697 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bzlgk\"" Jul 24 16:01:37.777767 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.777656 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-vk82w"] Jul 24 16:01:37.804782 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.804760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8g9\" (UniqueName: \"kubernetes.io/projected/c754a2c8-8694-41c0-8b4f-bedd0be27e9f-kube-api-access-7k8g9\") pod \"maas-controller-cc959d7c4-vk82w\" (UID: \"c754a2c8-8694-41c0-8b4f-bedd0be27e9f\") " pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:37.804892 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.804793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c754a2c8-8694-41c0-8b4f-bedd0be27e9f-webhook-cert\") pod \"maas-controller-cc959d7c4-vk82w\" (UID: \"c754a2c8-8694-41c0-8b4f-bedd0be27e9f\") " pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:37.905914 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.905877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c754a2c8-8694-41c0-8b4f-bedd0be27e9f-webhook-cert\") pod \"maas-controller-cc959d7c4-vk82w\" (UID: \"c754a2c8-8694-41c0-8b4f-bedd0be27e9f\") " pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:37.906030 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.906002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8g9\" (UniqueName: \"kubernetes.io/projected/c754a2c8-8694-41c0-8b4f-bedd0be27e9f-kube-api-access-7k8g9\") pod \"maas-controller-cc959d7c4-vk82w\" (UID: \"c754a2c8-8694-41c0-8b4f-bedd0be27e9f\") " pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:37.908415 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.908389 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c754a2c8-8694-41c0-8b4f-bedd0be27e9f-webhook-cert\") pod \"maas-controller-cc959d7c4-vk82w\" (UID: \"c754a2c8-8694-41c0-8b4f-bedd0be27e9f\") " pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:37.914484 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:37.914462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8g9\" (UniqueName: \"kubernetes.io/projected/c754a2c8-8694-41c0-8b4f-bedd0be27e9f-kube-api-access-7k8g9\") pod \"maas-controller-cc959d7c4-vk82w\" (UID: \"c754a2c8-8694-41c0-8b4f-bedd0be27e9f\") " pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:38.010005 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:38.009920 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8633ef8f-8873-4c60-9844-cff9b2a083ef" path="/var/lib/kubelet/pods/8633ef8f-8873-4c60-9844-cff9b2a083ef/volumes" Jul 24 16:01:38.080714 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:38.080685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:38.418437 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:38.418409 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cc959d7c4-vk82w"] Jul 24 16:01:38.420865 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:01:38.420834 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc754a2c8_8694_41c0_8b4f_bedd0be27e9f.slice/crio-f3d13a4277f5bf4ca67bc9fcb969974c4572f8e2db57a5269337beb62091aa91 WatchSource:0}: Error finding container f3d13a4277f5bf4ca67bc9fcb969974c4572f8e2db57a5269337beb62091aa91: Status 404 returned error can't find the container with id f3d13a4277f5bf4ca67bc9fcb969974c4572f8e2db57a5269337beb62091aa91 Jul 24 16:01:38.969922 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:38.969898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-vk82w" event={"ID":"c754a2c8-8694-41c0-8b4f-bedd0be27e9f","Type":"ContainerStarted","Data":"f3d13a4277f5bf4ca67bc9fcb969974c4572f8e2db57a5269337beb62091aa91"} Jul 24 16:01:39.976843 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:39.976810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cc959d7c4-vk82w" event={"ID":"c754a2c8-8694-41c0-8b4f-bedd0be27e9f","Type":"ContainerStarted","Data":"92f360d093b58e0f86eda7ae0ad1fbeaffb01f1dae2f0a6173da3d427378e396"} Jul 24 16:01:39.977255 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:39.976921 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:01:39.992730 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:39.992686 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cc959d7c4-vk82w" podStartSLOduration=2.4995483099999998 podStartE2EDuration="2.992671805s" totalImagesPullingTime="493.123495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:01:39.991577579 +0000 UTC m=+868.513186133" watchObservedRunningTime="2026-07-24 16:01:39.992671805 +0000 UTC m=+868.514280344" Jul 24 16:01:50.983838 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:01:50.983804 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cc959d7c4-vk82w" Jul 24 16:02:11.966553 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:02:11.966471 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:02:11.967133 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:02:11.966746 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:07:12.002535 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:07:12.002511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:07:12.004399 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:07:12.004373 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:12:12.032642 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:12:12.032619 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:12:12.036246 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:12:12.036224 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:14:51.659946 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.659902 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r"] Jul 24 16:14:51.663283 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.663267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.667180 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.667148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f991138c-serving-cert\"" Jul 24 16:14:51.700471 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.700446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r"] Jul 24 16:14:51.771299 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.771264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0e7050-28da-4344-b385-7efe79b779a0-openshift-service-ca\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.771464 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.771306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0e7050-28da-4344-b385-7efe79b779a0-maas-api-tls\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.771464 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.771327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tncq9\" (UniqueName: \"kubernetes.io/projected/0b0e7050-28da-4344-b385-7efe79b779a0-kube-api-access-tncq9\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.872019 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.871983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0e7050-28da-4344-b385-7efe79b779a0-openshift-service-ca\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.872184 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.872034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0e7050-28da-4344-b385-7efe79b779a0-maas-api-tls\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.872184 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.872052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tncq9\" (UniqueName: \"kubernetes.io/projected/0b0e7050-28da-4344-b385-7efe79b779a0-kube-api-access-tncq9\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.872636 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.872616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0e7050-28da-4344-b385-7efe79b779a0-openshift-service-ca\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.874730 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.874710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0e7050-28da-4344-b385-7efe79b779a0-maas-api-tls\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.880893 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.880868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tncq9\" (UniqueName: \"kubernetes.io/projected/0b0e7050-28da-4344-b385-7efe79b779a0-kube-api-access-tncq9\") pod \"maas-api-e2e-ait-f991138c-d67c88c58-pvd9r\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:51.973425 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:51.973356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:52.113647 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:52.113622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r"] Jul 24 16:14:52.116013 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:14:52.115982 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0e7050_28da_4344_b385_7efe79b779a0.slice/crio-c9d325a7345c362ccad66e67bdff1a129c3b2c459fa94687601334b2eef948e3 WatchSource:0}: Error finding container c9d325a7345c362ccad66e67bdff1a129c3b2c459fa94687601334b2eef948e3: Status 404 returned error can't find the container with id c9d325a7345c362ccad66e67bdff1a129c3b2c459fa94687601334b2eef948e3 Jul 24 16:14:52.117331 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:52.117314 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:14:52.717764 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:52.717737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" event={"ID":"0b0e7050-28da-4344-b385-7efe79b779a0","Type":"ContainerStarted","Data":"c9d325a7345c362ccad66e67bdff1a129c3b2c459fa94687601334b2eef948e3"} Jul 24 16:14:53.724743 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:53.724701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" event={"ID":"0b0e7050-28da-4344-b385-7efe79b779a0","Type":"ContainerStarted","Data":"d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc"} Jul 24 16:14:53.725179 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:53.724812 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:14:53.741492 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:53.741443 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" podStartSLOduration=2.163661515 podStartE2EDuration="2.741424884s" totalImagesPullingTime="577.763369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:14:53.7401372 +0000 UTC m=+1662.261745741" watchObservedRunningTime="2026-07-24 16:14:53.741424884 +0000 UTC m=+1662.263033423" Jul 24 16:14:59.735244 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:14:59.735216 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:15:27.464896 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.464860 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r"] Jul 24 16:15:27.467597 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.465131 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" podUID="0b0e7050-28da-4344-b385-7efe79b779a0" containerName="maas-api" containerID="cri-o://d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc" gracePeriod=30 Jul 24 16:15:27.711339 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.711314 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:15:27.813722 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.813698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0e7050-28da-4344-b385-7efe79b779a0-openshift-service-ca\") pod \"0b0e7050-28da-4344-b385-7efe79b779a0\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " Jul 24 16:15:27.813878 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.813756 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0e7050-28da-4344-b385-7efe79b779a0-maas-api-tls\") pod \"0b0e7050-28da-4344-b385-7efe79b779a0\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " Jul 24 16:15:27.813878 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.813783 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tncq9\" (UniqueName: \"kubernetes.io/projected/0b0e7050-28da-4344-b385-7efe79b779a0-kube-api-access-tncq9\") pod \"0b0e7050-28da-4344-b385-7efe79b779a0\" (UID: \"0b0e7050-28da-4344-b385-7efe79b779a0\") " Jul 24 16:15:27.814037 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.814015 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0e7050-28da-4344-b385-7efe79b779a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b0e7050-28da-4344-b385-7efe79b779a0" (UID: "0b0e7050-28da-4344-b385-7efe79b779a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:15:27.816102 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.816077 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0e7050-28da-4344-b385-7efe79b779a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b0e7050-28da-4344-b385-7efe79b779a0" (UID: "0b0e7050-28da-4344-b385-7efe79b779a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:15:27.816225 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.816111 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0e7050-28da-4344-b385-7efe79b779a0-kube-api-access-tncq9" (OuterVolumeSpecName: "kube-api-access-tncq9") pod "0b0e7050-28da-4344-b385-7efe79b779a0" (UID: "0b0e7050-28da-4344-b385-7efe79b779a0"). InnerVolumeSpecName "kube-api-access-tncq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:15:27.885926 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.885893 2580 generic.go:356] "Generic (PLEG): container finished" podID="0b0e7050-28da-4344-b385-7efe79b779a0" containerID="d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc" exitCode=0 Jul 24 16:15:27.886101 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.885956 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" Jul 24 16:15:27.886101 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.885989 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" event={"ID":"0b0e7050-28da-4344-b385-7efe79b779a0","Type":"ContainerDied","Data":"d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc"} Jul 24 16:15:27.886101 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.886049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r" event={"ID":"0b0e7050-28da-4344-b385-7efe79b779a0","Type":"ContainerDied","Data":"c9d325a7345c362ccad66e67bdff1a129c3b2c459fa94687601334b2eef948e3"} Jul 24 16:15:27.886101 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.886071 2580 scope.go:117] "RemoveContainer" containerID="d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc" Jul 24 16:15:27.894980 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.894923 2580 scope.go:117] "RemoveContainer" containerID="d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc" Jul 24 16:15:27.895227 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:15:27.895211 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc\": container with ID starting with d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc not found: ID does not exist" containerID="d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc" Jul 24 16:15:27.895282 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.895236 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc"} err="failed to get container status \"d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc\": rpc error: code = NotFound desc = could not find container \"d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc\": container with ID starting with d1c985d5d5863ec84a21251a42834a30ef50b04a03fb3e44dd08c44c83d2fabc not found: ID does not exist" Jul 24 16:15:27.906349 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.906325 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r"] Jul 24 16:15:27.909734 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.909715 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f991138c-d67c88c58-pvd9r"] Jul 24 16:15:27.915256 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.915239 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0e7050-28da-4344-b385-7efe79b779a0-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:15:27.915328 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.915259 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0e7050-28da-4344-b385-7efe79b779a0-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:15:27.915328 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:27.915268 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tncq9\" (UniqueName: \"kubernetes.io/projected/0b0e7050-28da-4344-b385-7efe79b779a0-kube-api-access-tncq9\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:15:28.010002 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:28.009980 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0e7050-28da-4344-b385-7efe79b779a0" path="/var/lib/kubelet/pods/0b0e7050-28da-4344-b385-7efe79b779a0/volumes" Jul 24 16:15:35.994457 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:35.994379 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f"] Jul 24 16:15:35.995052 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:35.995030 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0e7050-28da-4344-b385-7efe79b779a0" containerName="maas-api" Jul 24 16:15:35.995052 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:35.995051 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0e7050-28da-4344-b385-7efe79b779a0" containerName="maas-api" Jul 24 16:15:35.995186 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:35.995151 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0e7050-28da-4344-b385-7efe79b779a0" containerName="maas-api" Jul 24 16:15:35.999608 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:35.999590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.002819 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.002798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0b92f583-serving-cert\"" Jul 24 16:15:36.015068 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.015044 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f"] Jul 24 16:15:36.082170 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.082139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz747\" (UniqueName: \"kubernetes.io/projected/04dbc016-6fbd-48f1-a80f-dfde4670f15e-kube-api-access-cz747\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.082295 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.082270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04dbc016-6fbd-48f1-a80f-dfde4670f15e-maas-api-tls\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.082363 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.082345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04dbc016-6fbd-48f1-a80f-dfde4670f15e-openshift-service-ca\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.183464 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.183433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04dbc016-6fbd-48f1-a80f-dfde4670f15e-maas-api-tls\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.183586 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.183490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04dbc016-6fbd-48f1-a80f-dfde4670f15e-openshift-service-ca\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.183586 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.183572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz747\" (UniqueName: \"kubernetes.io/projected/04dbc016-6fbd-48f1-a80f-dfde4670f15e-kube-api-access-cz747\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.184173 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.184152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04dbc016-6fbd-48f1-a80f-dfde4670f15e-openshift-service-ca\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.185989 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.185969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04dbc016-6fbd-48f1-a80f-dfde4670f15e-maas-api-tls\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.192161 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.192137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz747\" (UniqueName: \"kubernetes.io/projected/04dbc016-6fbd-48f1-a80f-dfde4670f15e-kube-api-access-cz747\") pod \"maas-api-e2e-migrate-0b92f583-57595b9478-22c6f\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.310138 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.310111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:36.673656 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.673632 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f"] Jul 24 16:15:36.676089 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:15:36.676060 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dbc016_6fbd_48f1_a80f_dfde4670f15e.slice/crio-d0468f732bfb049cca13b7a98136f9ddf598086b0967108a4de4b50b911966e4 WatchSource:0}: Error finding container d0468f732bfb049cca13b7a98136f9ddf598086b0967108a4de4b50b911966e4: Status 404 returned error can't find the container with id d0468f732bfb049cca13b7a98136f9ddf598086b0967108a4de4b50b911966e4 Jul 24 16:15:36.930108 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:36.930031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" event={"ID":"04dbc016-6fbd-48f1-a80f-dfde4670f15e","Type":"ContainerStarted","Data":"d0468f732bfb049cca13b7a98136f9ddf598086b0967108a4de4b50b911966e4"} Jul 24 16:15:37.936670 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:37.936629 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" event={"ID":"04dbc016-6fbd-48f1-a80f-dfde4670f15e","Type":"ContainerStarted","Data":"2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8"} Jul 24 16:15:37.937081 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:37.936693 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:15:37.953233 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:37.953180 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" podStartSLOduration=2.637872877 podStartE2EDuration="2.953164301s" totalImagesPullingTime="315.291424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:15:37.952413918 +0000 UTC m=+1706.474022459" watchObservedRunningTime="2026-07-24 16:15:37.953164301 +0000 UTC m=+1706.474772840" Jul 24 16:15:43.948003 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:15:43.947975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:16:11.217622 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.217585 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f"] Jul 24 16:16:11.218209 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.217801 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" podUID="04dbc016-6fbd-48f1-a80f-dfde4670f15e" containerName="maas-api" containerID="cri-o://2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8" gracePeriod=30 Jul 24 16:16:11.462770 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.462747 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:16:11.513175 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.513148 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz747\" (UniqueName: \"kubernetes.io/projected/04dbc016-6fbd-48f1-a80f-dfde4670f15e-kube-api-access-cz747\") pod \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " Jul 24 16:16:11.513364 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.513207 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04dbc016-6fbd-48f1-a80f-dfde4670f15e-openshift-service-ca\") pod \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " Jul 24 16:16:11.513364 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.513240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04dbc016-6fbd-48f1-a80f-dfde4670f15e-maas-api-tls\") pod \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\" (UID: \"04dbc016-6fbd-48f1-a80f-dfde4670f15e\") " Jul 24 16:16:11.513609 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.513585 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04dbc016-6fbd-48f1-a80f-dfde4670f15e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04dbc016-6fbd-48f1-a80f-dfde4670f15e" (UID: "04dbc016-6fbd-48f1-a80f-dfde4670f15e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:16:11.515480 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.515454 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04dbc016-6fbd-48f1-a80f-dfde4670f15e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04dbc016-6fbd-48f1-a80f-dfde4670f15e" (UID: "04dbc016-6fbd-48f1-a80f-dfde4670f15e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:16:11.515480 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.515460 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dbc016-6fbd-48f1-a80f-dfde4670f15e-kube-api-access-cz747" (OuterVolumeSpecName: "kube-api-access-cz747") pod "04dbc016-6fbd-48f1-a80f-dfde4670f15e" (UID: "04dbc016-6fbd-48f1-a80f-dfde4670f15e"). InnerVolumeSpecName "kube-api-access-cz747". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:16:11.614862 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.614825 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz747\" (UniqueName: \"kubernetes.io/projected/04dbc016-6fbd-48f1-a80f-dfde4670f15e-kube-api-access-cz747\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:16:11.614862 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.614855 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04dbc016-6fbd-48f1-a80f-dfde4670f15e-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:16:11.614862 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:11.614869 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04dbc016-6fbd-48f1-a80f-dfde4670f15e-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:16:12.105995 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.105961 2580 generic.go:356] "Generic (PLEG): container finished" podID="04dbc016-6fbd-48f1-a80f-dfde4670f15e" containerID="2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8" exitCode=0 Jul 24 16:16:12.106168 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.106005 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" Jul 24 16:16:12.106168 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.106017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" event={"ID":"04dbc016-6fbd-48f1-a80f-dfde4670f15e","Type":"ContainerDied","Data":"2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8"} Jul 24 16:16:12.106168 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.106083 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f" event={"ID":"04dbc016-6fbd-48f1-a80f-dfde4670f15e","Type":"ContainerDied","Data":"d0468f732bfb049cca13b7a98136f9ddf598086b0967108a4de4b50b911966e4"} Jul 24 16:16:12.106168 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.106108 2580 scope.go:117] "RemoveContainer" containerID="2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8" Jul 24 16:16:12.122114 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.122094 2580 scope.go:117] "RemoveContainer" containerID="2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8" Jul 24 16:16:12.122392 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:16:12.122370 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8\": container with ID starting with 2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8 not found: ID does not exist" containerID="2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8" Jul 24 16:16:12.122455 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.122399 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8"} err="failed to get container status \"2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8\": rpc error: code = NotFound desc = could not find container \"2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8\": container with ID starting with 2262cc620a47e9544834b672de964366a310881f018ca19566cedc689a0b86e8 not found: ID does not exist" Jul 24 16:16:12.125651 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.125621 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f"] Jul 24 16:16:12.129514 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:12.129493 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0b92f583-57595b9478-22c6f"] Jul 24 16:16:14.010608 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:14.010574 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04dbc016-6fbd-48f1-a80f-dfde4670f15e" path="/var/lib/kubelet/pods/04dbc016-6fbd-48f1-a80f-dfde4670f15e/volumes" Jul 24 16:16:19.179436 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.179402 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr"] Jul 24 16:16:19.179928 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.179841 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04dbc016-6fbd-48f1-a80f-dfde4670f15e" containerName="maas-api" Jul 24 16:16:19.179928 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.179857 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04dbc016-6fbd-48f1-a80f-dfde4670f15e" containerName="maas-api" Jul 24 16:16:19.180076 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.179955 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04dbc016-6fbd-48f1-a80f-dfde4670f15e" containerName="maas-api" Jul 24 16:16:19.188075 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.188043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.191622 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.190849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2cc8eb19-serving-cert\"" Jul 24 16:16:19.193091 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.193055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr"] Jul 24 16:16:19.285416 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.285381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be86d7e3-a081-435e-b580-7798a84094ff-openshift-service-ca\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.285594 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.285440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.285594 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.285509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdw46\" (UniqueName: \"kubernetes.io/projected/be86d7e3-a081-435e-b580-7798a84094ff-kube-api-access-tdw46\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.386535 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.386490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be86d7e3-a081-435e-b580-7798a84094ff-openshift-service-ca\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.386712 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.386552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.386712 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.386580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdw46\" (UniqueName: \"kubernetes.io/projected/be86d7e3-a081-435e-b580-7798a84094ff-kube-api-access-tdw46\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.386803 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:16:19.386712 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-serving-cert: secret "maas-api-e2e-ait-2cc8eb19-serving-cert" not found Jul 24 16:16:19.386803 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:16:19.386785 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls podName:be86d7e3-a081-435e-b580-7798a84094ff nodeName:}" failed. No retries permitted until 2026-07-24 16:16:19.886768535 +0000 UTC m=+1748.408377052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls") pod "maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" (UID: "be86d7e3-a081-435e-b580-7798a84094ff") : secret "maas-api-e2e-ait-2cc8eb19-serving-cert" not found Jul 24 16:16:19.387177 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.387159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be86d7e3-a081-435e-b580-7798a84094ff-openshift-service-ca\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.401682 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.401661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdw46\" (UniqueName: \"kubernetes.io/projected/be86d7e3-a081-435e-b580-7798a84094ff-kube-api-access-tdw46\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.891684 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.891646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:19.894396 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:19.894370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls\") pod \"maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:20.102288 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:20.102252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:20.243662 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:20.243635 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr"] Jul 24 16:16:20.246167 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:16:20.246141 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe86d7e3_a081_435e_b580_7798a84094ff.slice/crio-bdda2f36f870e395a7958a731b8e065e83b31c8a5d6a252ab6cc7887b1ddbe82 WatchSource:0}: Error finding container bdda2f36f870e395a7958a731b8e065e83b31c8a5d6a252ab6cc7887b1ddbe82: Status 404 returned error can't find the container with id bdda2f36f870e395a7958a731b8e065e83b31c8a5d6a252ab6cc7887b1ddbe82 Jul 24 16:16:21.152771 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:21.152740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" event={"ID":"be86d7e3-a081-435e-b580-7798a84094ff","Type":"ContainerStarted","Data":"e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4"} Jul 24 16:16:21.152961 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:21.152788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" event={"ID":"be86d7e3-a081-435e-b580-7798a84094ff","Type":"ContainerStarted","Data":"bdda2f36f870e395a7958a731b8e065e83b31c8a5d6a252ab6cc7887b1ddbe82"} Jul 24 16:16:21.152961 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:21.152894 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:21.170804 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:21.170760 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" podStartSLOduration=1.490530026 podStartE2EDuration="2.170745179s" totalImagesPullingTime="680.215153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:16:21.168841359 +0000 UTC m=+1749.690449927" watchObservedRunningTime="2026-07-24 16:16:21.170745179 +0000 UTC m=+1749.692353718" Jul 24 16:16:27.163540 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:27.163512 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:55.214277 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.214242 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr"] Jul 24 16:16:55.214759 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.214461 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" podUID="be86d7e3-a081-435e-b580-7798a84094ff" containerName="maas-api" containerID="cri-o://e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4" gracePeriod=30 Jul 24 16:16:55.474718 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.474669 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:55.609141 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.609112 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdw46\" (UniqueName: \"kubernetes.io/projected/be86d7e3-a081-435e-b580-7798a84094ff-kube-api-access-tdw46\") pod \"be86d7e3-a081-435e-b580-7798a84094ff\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " Jul 24 16:16:55.609293 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.609182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls\") pod \"be86d7e3-a081-435e-b580-7798a84094ff\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " Jul 24 16:16:55.609293 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.609235 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be86d7e3-a081-435e-b580-7798a84094ff-openshift-service-ca\") pod \"be86d7e3-a081-435e-b580-7798a84094ff\" (UID: \"be86d7e3-a081-435e-b580-7798a84094ff\") " Jul 24 16:16:55.609607 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.609580 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be86d7e3-a081-435e-b580-7798a84094ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be86d7e3-a081-435e-b580-7798a84094ff" (UID: "be86d7e3-a081-435e-b580-7798a84094ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:16:55.611329 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.611306 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be86d7e3-a081-435e-b580-7798a84094ff" (UID: "be86d7e3-a081-435e-b580-7798a84094ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:16:55.611415 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.611329 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be86d7e3-a081-435e-b580-7798a84094ff-kube-api-access-tdw46" (OuterVolumeSpecName: "kube-api-access-tdw46") pod "be86d7e3-a081-435e-b580-7798a84094ff" (UID: "be86d7e3-a081-435e-b580-7798a84094ff"). InnerVolumeSpecName "kube-api-access-tdw46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:16:55.710770 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.710724 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdw46\" (UniqueName: \"kubernetes.io/projected/be86d7e3-a081-435e-b580-7798a84094ff-kube-api-access-tdw46\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:16:55.710770 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.710766 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be86d7e3-a081-435e-b580-7798a84094ff-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:16:55.710770 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:55.710778 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be86d7e3-a081-435e-b580-7798a84094ff-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:16:56.322633 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.322546 2580 generic.go:356] "Generic (PLEG): container finished" podID="be86d7e3-a081-435e-b580-7798a84094ff" containerID="e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4" exitCode=0 Jul 24 16:16:56.322633 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.322591 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" Jul 24 16:16:56.322633 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.322604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" event={"ID":"be86d7e3-a081-435e-b580-7798a84094ff","Type":"ContainerDied","Data":"e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4"} Jul 24 16:16:56.323179 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.322655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr" event={"ID":"be86d7e3-a081-435e-b580-7798a84094ff","Type":"ContainerDied","Data":"bdda2f36f870e395a7958a731b8e065e83b31c8a5d6a252ab6cc7887b1ddbe82"} Jul 24 16:16:56.323179 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.322685 2580 scope.go:117] "RemoveContainer" containerID="e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4" Jul 24 16:16:56.331722 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.331696 2580 scope.go:117] "RemoveContainer" containerID="e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4" Jul 24 16:16:56.331980 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:16:56.331956 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4\": container with ID starting with e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4 not found: ID does not exist" containerID="e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4" Jul 24 16:16:56.332031 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.331988 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4"} err="failed to get container status \"e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4\": rpc error: code = NotFound desc = could not find container \"e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4\": container with ID starting with e4946ab6e75cda685ff118f04671b2495d30cee66fcf08ea0ef171763a36f6b4 not found: ID does not exist" Jul 24 16:16:56.339484 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.339464 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr"] Jul 24 16:16:56.343645 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:56.343626 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cc8eb19-58c58f5fdf-gjdkr"] Jul 24 16:16:58.010811 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:16:58.010780 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be86d7e3-a081-435e-b580-7798a84094ff" path="/var/lib/kubelet/pods/be86d7e3-a081-435e-b580-7798a84094ff/volumes" Jul 24 16:17:04.387539 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.387463 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp"] Jul 24 16:17:04.388085 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.388067 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be86d7e3-a081-435e-b580-7798a84094ff" containerName="maas-api" Jul 24 16:17:04.388137 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.388089 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be86d7e3-a081-435e-b580-7798a84094ff" containerName="maas-api" Jul 24 16:17:04.388345 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.388208 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be86d7e3-a081-435e-b580-7798a84094ff" containerName="maas-api" Jul 24 16:17:04.393052 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.393029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.394968 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.394947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8426fff8-serving-cert\"" Jul 24 16:17:04.399105 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.399084 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp"] Jul 24 16:17:04.486172 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.486122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-openshift-service-ca\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.486374 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.486235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxd5\" (UniqueName: \"kubernetes.io/projected/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-kube-api-access-7cxd5\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.486374 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.486264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.587452 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.587411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxd5\" (UniqueName: \"kubernetes.io/projected/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-kube-api-access-7cxd5\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.587617 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.587457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.587617 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.587544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-openshift-service-ca\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.587702 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:17:04.587628 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-serving-cert: secret "maas-api-e2e-derive-8426fff8-serving-cert" not found Jul 24 16:17:04.587702 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:17:04.587686 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls podName:f387a1e8-b1a9-44c0-a6ca-69a6ad64546d nodeName:}" failed. No retries permitted until 2026-07-24 16:17:05.087669925 +0000 UTC m=+1793.609278443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls") pod "maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" (UID: "f387a1e8-b1a9-44c0-a6ca-69a6ad64546d") : secret "maas-api-e2e-derive-8426fff8-serving-cert" not found Jul 24 16:17:04.588210 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.588193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-openshift-service-ca\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:04.598576 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:04.598557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxd5\" (UniqueName: \"kubernetes.io/projected/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-kube-api-access-7cxd5\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:05.092183 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:05.092149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:05.094710 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:05.094679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls\") pod \"maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:05.305454 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:05.305418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:05.450553 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:05.450523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp"] Jul 24 16:17:05.453523 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:17:05.453490 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf387a1e8_b1a9_44c0_a6ca_69a6ad64546d.slice/crio-c1bd213d80a61eefd74cc581b340dd4e80623c026f1896319bb91cff6bb79603 WatchSource:0}: Error finding container c1bd213d80a61eefd74cc581b340dd4e80623c026f1896319bb91cff6bb79603: Status 404 returned error can't find the container with id c1bd213d80a61eefd74cc581b340dd4e80623c026f1896319bb91cff6bb79603 Jul 24 16:17:06.377359 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:06.377259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" event={"ID":"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d","Type":"ContainerStarted","Data":"3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035"} Jul 24 16:17:06.377359 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:06.377310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" event={"ID":"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d","Type":"ContainerStarted","Data":"c1bd213d80a61eefd74cc581b340dd4e80623c026f1896319bb91cff6bb79603"} Jul 24 16:17:06.377586 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:06.377397 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:06.405778 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:06.405725 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" podStartSLOduration=1.8049370649999998 podStartE2EDuration="2.405708633s" totalImagesPullingTime="600.771568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:17:06.404089719 +0000 UTC m=+1794.925698260" watchObservedRunningTime="2026-07-24 16:17:06.405708633 +0000 UTC m=+1794.927317172" Jul 24 16:17:12.063282 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:12.063245 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:17:12.068171 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:12.068147 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:17:12.387837 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:12.387740 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:39.638298 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.638261 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp"] Jul 24 16:17:39.638884 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.638500 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" podUID="f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" containerName="maas-api" containerID="cri-o://3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035" gracePeriod=30 Jul 24 16:17:39.884705 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.884683 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:39.983299 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.983235 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-openshift-service-ca\") pod \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " Jul 24 16:17:39.983299 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.983282 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cxd5\" (UniqueName: \"kubernetes.io/projected/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-kube-api-access-7cxd5\") pod \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " Jul 24 16:17:39.983451 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.983325 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls\") pod \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\" (UID: \"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d\") " Jul 24 16:17:39.983606 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.983584 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" (UID: "f387a1e8-b1a9-44c0-a6ca-69a6ad64546d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:17:39.985410 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.985385 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" (UID: "f387a1e8-b1a9-44c0-a6ca-69a6ad64546d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:17:39.985518 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:39.985448 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-kube-api-access-7cxd5" (OuterVolumeSpecName: "kube-api-access-7cxd5") pod "f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" (UID: "f387a1e8-b1a9-44c0-a6ca-69a6ad64546d"). InnerVolumeSpecName "kube-api-access-7cxd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:17:40.084956 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.084907 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:17:40.084956 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.084930 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxd5\" (UniqueName: \"kubernetes.io/projected/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-kube-api-access-7cxd5\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:17:40.085108 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.084964 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:17:40.535615 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.535580 2580 generic.go:356] "Generic (PLEG): container finished" podID="f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" containerID="3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035" exitCode=0 Jul 24 16:17:40.535809 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.535632 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" Jul 24 16:17:40.535809 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.535661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" event={"ID":"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d","Type":"ContainerDied","Data":"3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035"} Jul 24 16:17:40.535809 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.535764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp" event={"ID":"f387a1e8-b1a9-44c0-a6ca-69a6ad64546d","Type":"ContainerDied","Data":"c1bd213d80a61eefd74cc581b340dd4e80623c026f1896319bb91cff6bb79603"} Jul 24 16:17:40.535809 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.535792 2580 scope.go:117] "RemoveContainer" containerID="3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035" Jul 24 16:17:40.544303 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.544283 2580 scope.go:117] "RemoveContainer" containerID="3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035" Jul 24 16:17:40.544557 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:17:40.544540 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035\": container with ID starting with 3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035 not found: ID does not exist" containerID="3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035" Jul 24 16:17:40.544611 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.544565 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035"} err="failed to get container status \"3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035\": rpc error: code = NotFound desc = could not find container \"3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035\": container with ID starting with 3eea066a822400d8eb3c6c0b71f494c154a35a18de5fd16b8018fe25366ce035 not found: ID does not exist" Jul 24 16:17:40.553419 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.553393 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp"] Jul 24 16:17:40.555019 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:40.555001 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8426fff8-6dc5bc4fdc-6cjbp"] Jul 24 16:17:42.011171 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:17:42.011140 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" path="/var/lib/kubelet/pods/f387a1e8-b1a9-44c0-a6ca-69a6ad64546d/volumes" Jul 24 16:19:24.599270 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.599233 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm"] Jul 24 16:19:24.599840 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.599824 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" containerName="maas-api" Jul 24 16:19:24.599889 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.599843 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" containerName="maas-api" Jul 24 16:19:24.599967 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.599955 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f387a1e8-b1a9-44c0-a6ca-69a6ad64546d" containerName="maas-api" Jul 24 16:19:24.603052 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.603029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.605183 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.605158 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e1fef210-serving-cert\"" Jul 24 16:19:24.613083 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.613057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm"] Jul 24 16:19:24.726791 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.726758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.726988 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.726882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzplv\" (UniqueName: \"kubernetes.io/projected/a870772c-6791-4997-82b7-1ab226fc6a69-kube-api-access-jzplv\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.726988 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.726967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870772c-6791-4997-82b7-1ab226fc6a69-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.828304 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.828261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzplv\" (UniqueName: \"kubernetes.io/projected/a870772c-6791-4997-82b7-1ab226fc6a69-kube-api-access-jzplv\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.828482 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.828314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870772c-6791-4997-82b7-1ab226fc6a69-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.828482 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.828353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.828482 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:19:24.828450 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-serving-cert: secret "maas-api-e2e-mt-e1fef210-serving-cert" not found Jul 24 16:19:24.828620 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:19:24.828504 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls podName:a870772c-6791-4997-82b7-1ab226fc6a69 nodeName:}" failed. No retries permitted until 2026-07-24 16:19:25.328488561 +0000 UTC m=+1933.850097079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls") pod "maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" (UID: "a870772c-6791-4997-82b7-1ab226fc6a69") : secret "maas-api-e2e-mt-e1fef210-serving-cert" not found Jul 24 16:19:24.829041 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.829022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870772c-6791-4997-82b7-1ab226fc6a69-openshift-service-ca\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:24.836479 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:24.836457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzplv\" (UniqueName: \"kubernetes.io/projected/a870772c-6791-4997-82b7-1ab226fc6a69-kube-api-access-jzplv\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:25.333316 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:25.333282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:25.336040 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:25.336012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls\") pod \"maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:25.514110 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:25.514071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:25.650300 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:25.650273 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm"] Jul 24 16:19:25.654029 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:19:25.653995 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda870772c_6791_4997_82b7_1ab226fc6a69.slice/crio-dd363f1aca765bac8b5c9a6ffead64eb3762a024752ca603d13fbb5d103231be WatchSource:0}: Error finding container dd363f1aca765bac8b5c9a6ffead64eb3762a024752ca603d13fbb5d103231be: Status 404 returned error can't find the container with id dd363f1aca765bac8b5c9a6ffead64eb3762a024752ca603d13fbb5d103231be Jul 24 16:19:26.033088 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:26.033055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" event={"ID":"a870772c-6791-4997-82b7-1ab226fc6a69","Type":"ContainerStarted","Data":"dd363f1aca765bac8b5c9a6ffead64eb3762a024752ca603d13fbb5d103231be"} Jul 24 16:19:27.039258 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:27.039224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" event={"ID":"a870772c-6791-4997-82b7-1ab226fc6a69","Type":"ContainerStarted","Data":"a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f"} Jul 24 16:19:27.039639 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:27.039337 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:27.063081 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:27.059920 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" podStartSLOduration=2.5187593440000002 podStartE2EDuration="3.059901303s" totalImagesPullingTime="541.141959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:19:27.057004646 +0000 UTC m=+1935.578613186" watchObservedRunningTime="2026-07-24 16:19:27.059901303 +0000 UTC m=+1935.581509845" Jul 24 16:19:33.050697 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:33.050667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:46.694864 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:46.694831 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm"] Jul 24 16:19:46.695321 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:46.695075 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" podUID="a870772c-6791-4997-82b7-1ab226fc6a69" containerName="maas-api" containerID="cri-o://a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f" gracePeriod=30 Jul 24 16:19:46.946580 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:46.946523 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:47.029018 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.028989 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls\") pod \"a870772c-6791-4997-82b7-1ab226fc6a69\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " Jul 24 16:19:47.029171 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.029024 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzplv\" (UniqueName: \"kubernetes.io/projected/a870772c-6791-4997-82b7-1ab226fc6a69-kube-api-access-jzplv\") pod \"a870772c-6791-4997-82b7-1ab226fc6a69\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " Jul 24 16:19:47.029171 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.029060 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870772c-6791-4997-82b7-1ab226fc6a69-openshift-service-ca\") pod \"a870772c-6791-4997-82b7-1ab226fc6a69\" (UID: \"a870772c-6791-4997-82b7-1ab226fc6a69\") " Jul 24 16:19:47.029578 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.029539 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a870772c-6791-4997-82b7-1ab226fc6a69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a870772c-6791-4997-82b7-1ab226fc6a69" (UID: "a870772c-6791-4997-82b7-1ab226fc6a69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:19:47.031246 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.031225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a870772c-6791-4997-82b7-1ab226fc6a69-kube-api-access-jzplv" (OuterVolumeSpecName: "kube-api-access-jzplv") pod "a870772c-6791-4997-82b7-1ab226fc6a69" (UID: "a870772c-6791-4997-82b7-1ab226fc6a69"). InnerVolumeSpecName "kube-api-access-jzplv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:47.031342 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.031225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a870772c-6791-4997-82b7-1ab226fc6a69" (UID: "a870772c-6791-4997-82b7-1ab226fc6a69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:19:47.129902 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.129873 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a870772c-6791-4997-82b7-1ab226fc6a69-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:19:47.129902 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.129899 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzplv\" (UniqueName: \"kubernetes.io/projected/a870772c-6791-4997-82b7-1ab226fc6a69-kube-api-access-jzplv\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:19:47.130124 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.129910 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870772c-6791-4997-82b7-1ab226fc6a69-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:19:47.132104 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.132070 2580 generic.go:356] "Generic (PLEG): container finished" podID="a870772c-6791-4997-82b7-1ab226fc6a69" containerID="a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f" exitCode=0 Jul 24 16:19:47.132229 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.132110 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" Jul 24 16:19:47.132229 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.132161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" event={"ID":"a870772c-6791-4997-82b7-1ab226fc6a69","Type":"ContainerDied","Data":"a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f"} Jul 24 16:19:47.132229 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.132212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm" event={"ID":"a870772c-6791-4997-82b7-1ab226fc6a69","Type":"ContainerDied","Data":"dd363f1aca765bac8b5c9a6ffead64eb3762a024752ca603d13fbb5d103231be"} Jul 24 16:19:47.132335 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.132242 2580 scope.go:117] "RemoveContainer" containerID="a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f" Jul 24 16:19:47.148736 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.148720 2580 scope.go:117] "RemoveContainer" containerID="a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f" Jul 24 16:19:47.149044 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:19:47.149025 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f\": container with ID starting with a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f not found: ID does not exist" containerID="a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f" Jul 24 16:19:47.149089 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.149052 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f"} err="failed to get container status \"a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f\": rpc error: code = NotFound desc = could not find container \"a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f\": container with ID starting with a7338fad97ea4891aefd5f8b156d7f447d22054c25727587eda75d8eedb2ee8f not found: ID does not exist" Jul 24 16:19:47.161218 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.161196 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm"] Jul 24 16:19:47.165075 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:47.165046 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1fef210-7d7d6cb777-s96dm"] Jul 24 16:19:48.010263 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:19:48.010230 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a870772c-6791-4997-82b7-1ab226fc6a69" path="/var/lib/kubelet/pods/a870772c-6791-4997-82b7-1ab226fc6a69/volumes" Jul 24 16:20:19.885100 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:19.885061 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6"] Jul 24 16:20:19.885709 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:19.885690 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a870772c-6791-4997-82b7-1ab226fc6a69" containerName="maas-api" Jul 24 16:20:19.885769 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:19.885714 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a870772c-6791-4997-82b7-1ab226fc6a69" containerName="maas-api" Jul 24 16:20:19.885845 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:19.885833 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a870772c-6791-4997-82b7-1ab226fc6a69" containerName="maas-api" Jul 24 16:20:19.889408 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:19.889387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:19.891873 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:19.891851 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b13121e3-serving-cert\"" Jul 24 16:20:19.898745 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:19.898716 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6"] Jul 24 16:20:20.021801 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.021766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa0f01-f119-49ba-9e15-dca36cce3faf-openshift-service-ca\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.021801 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.021799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9l6\" (UniqueName: \"kubernetes.io/projected/e0aa0f01-f119-49ba-9e15-dca36cce3faf-kube-api-access-ld9l6\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.022043 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.021916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.123359 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.123323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.123536 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.123400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa0f01-f119-49ba-9e15-dca36cce3faf-openshift-service-ca\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.123536 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.123418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9l6\" (UniqueName: \"kubernetes.io/projected/e0aa0f01-f119-49ba-9e15-dca36cce3faf-kube-api-access-ld9l6\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.123536 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:20:20.123498 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-serving-cert: secret "maas-api-e2e-mt-b13121e3-serving-cert" not found Jul 24 16:20:20.123644 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:20:20.123587 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls podName:e0aa0f01-f119-49ba-9e15-dca36cce3faf nodeName:}" failed. No retries permitted until 2026-07-24 16:20:20.62356402 +0000 UTC m=+1989.145172554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls") pod "maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" (UID: "e0aa0f01-f119-49ba-9e15-dca36cce3faf") : secret "maas-api-e2e-mt-b13121e3-serving-cert" not found Jul 24 16:20:20.124150 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.124130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa0f01-f119-49ba-9e15-dca36cce3faf-openshift-service-ca\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.132298 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.132280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9l6\" (UniqueName: \"kubernetes.io/projected/e0aa0f01-f119-49ba-9e15-dca36cce3faf-kube-api-access-ld9l6\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.628138 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:20.628088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:20.628315 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:20:20.628234 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-serving-cert: secret "maas-api-e2e-mt-b13121e3-serving-cert" not found Jul 24 16:20:20.628315 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:20:20.628300 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls podName:e0aa0f01-f119-49ba-9e15-dca36cce3faf nodeName:}" failed. No retries permitted until 2026-07-24 16:20:21.628284859 +0000 UTC m=+1990.149893380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls") pod "maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" (UID: "e0aa0f01-f119-49ba-9e15-dca36cce3faf") : secret "maas-api-e2e-mt-b13121e3-serving-cert" not found Jul 24 16:20:21.636536 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:21.636502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:21.639171 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:21.639146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls\") pod \"maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:21.701149 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:21.701113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:21.843854 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:21.843828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6"] Jul 24 16:20:21.846418 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:20:21.846382 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aa0f01_f119_49ba_9e15_dca36cce3faf.slice/crio-323a8b78a505e15a8d10f299b2c4c534a4c98be71a9ef2be5548463eaa69ff09 WatchSource:0}: Error finding container 323a8b78a505e15a8d10f299b2c4c534a4c98be71a9ef2be5548463eaa69ff09: Status 404 returned error can't find the container with id 323a8b78a505e15a8d10f299b2c4c534a4c98be71a9ef2be5548463eaa69ff09 Jul 24 16:20:21.847929 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:21.847912 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:20:22.302656 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:22.302626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" event={"ID":"e0aa0f01-f119-49ba-9e15-dca36cce3faf","Type":"ContainerStarted","Data":"323a8b78a505e15a8d10f299b2c4c534a4c98be71a9ef2be5548463eaa69ff09"} Jul 24 16:20:23.308807 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:23.308771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" event={"ID":"e0aa0f01-f119-49ba-9e15-dca36cce3faf","Type":"ContainerStarted","Data":"b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45"} Jul 24 16:20:23.309255 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:23.308883 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:20:23.325606 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:23.325555 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" podStartSLOduration=3.936188915 podStartE2EDuration="4.325538513s" totalImagesPullingTime="389.349598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:20:23.324585763 +0000 UTC m=+1991.846194306" watchObservedRunningTime="2026-07-24 16:20:23.325538513 +0000 UTC m=+1991.847147053" Jul 24 16:20:29.318912 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:20:29.318881 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:21:14.124888 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.124840 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6"] Jul 24 16:21:14.125502 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.125117 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" podUID="e0aa0f01-f119-49ba-9e15-dca36cce3faf" containerName="maas-api" containerID="cri-o://b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45" gracePeriod=30 Jul 24 16:21:14.383223 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.383161 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:21:14.427556 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.427517 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld9l6\" (UniqueName: \"kubernetes.io/projected/e0aa0f01-f119-49ba-9e15-dca36cce3faf-kube-api-access-ld9l6\") pod \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " Jul 24 16:21:14.427760 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.427606 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls\") pod \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " Jul 24 16:21:14.427760 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.427638 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa0f01-f119-49ba-9e15-dca36cce3faf-openshift-service-ca\") pod \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\" (UID: \"e0aa0f01-f119-49ba-9e15-dca36cce3faf\") " Jul 24 16:21:14.428061 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.428026 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0aa0f01-f119-49ba-9e15-dca36cce3faf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0aa0f01-f119-49ba-9e15-dca36cce3faf" (UID: "e0aa0f01-f119-49ba-9e15-dca36cce3faf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:21:14.429866 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.429840 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0aa0f01-f119-49ba-9e15-dca36cce3faf" (UID: "e0aa0f01-f119-49ba-9e15-dca36cce3faf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:21:14.429866 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.429845 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aa0f01-f119-49ba-9e15-dca36cce3faf-kube-api-access-ld9l6" (OuterVolumeSpecName: "kube-api-access-ld9l6") pod "e0aa0f01-f119-49ba-9e15-dca36cce3faf" (UID: "e0aa0f01-f119-49ba-9e15-dca36cce3faf"). InnerVolumeSpecName "kube-api-access-ld9l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:21:14.528364 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.528329 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aa0f01-f119-49ba-9e15-dca36cce3faf-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.528364 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.528361 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa0f01-f119-49ba-9e15-dca36cce3faf-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.528364 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.528372 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9l6\" (UniqueName: \"kubernetes.io/projected/e0aa0f01-f119-49ba-9e15-dca36cce3faf-kube-api-access-ld9l6\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.551119 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.551089 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0aa0f01-f119-49ba-9e15-dca36cce3faf" containerID="b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45" exitCode=0 Jul 24 16:21:14.551316 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.551141 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" Jul 24 16:21:14.551316 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.551188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" event={"ID":"e0aa0f01-f119-49ba-9e15-dca36cce3faf","Type":"ContainerDied","Data":"b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45"} Jul 24 16:21:14.551316 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.551248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" event={"ID":"e0aa0f01-f119-49ba-9e15-dca36cce3faf","Type":"ContainerDied","Data":"323a8b78a505e15a8d10f299b2c4c534a4c98be71a9ef2be5548463eaa69ff09"} Jul 24 16:21:14.551316 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.551277 2580 scope.go:117] "RemoveContainer" containerID="b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45" Jul 24 16:21:14.560769 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.560750 2580 scope.go:117] "RemoveContainer" containerID="b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45" Jul 24 16:21:14.561032 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:21:14.561011 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45\": container with ID starting with b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45 not found: ID does not exist" containerID="b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45" Jul 24 16:21:14.561101 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.561044 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45"} err="failed to get container status \"b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45\": rpc error: code = NotFound desc = could not find container \"b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45\": container with ID starting with b73009883d576bbb3e0599a06c54316725cfc7b83cb1af1212c7da5898804e45 not found: ID does not exist" Jul 24 16:21:14.572447 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.572421 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6"] Jul 24 16:21:14.574241 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:14.574223 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6"] Jul 24 16:21:16.010864 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:16.010828 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0aa0f01-f119-49ba-9e15-dca36cce3faf" path="/var/lib/kubelet/pods/e0aa0f01-f119-49ba-9e15-dca36cce3faf/volumes" Jul 24 16:21:17.314520 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:21:17.314450 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b13121e3-7d695f8d47-67ks6" podUID="e0aa0f01-f119-49ba-9e15-dca36cce3faf" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.47:8443/health\": context deadline exceeded" Jul 24 16:22:12.094377 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:22:12.094346 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:22:12.100267 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:22:12.100238 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:23:08.765144 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.765056 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c"] Jul 24 16:23:08.765723 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.765670 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0aa0f01-f119-49ba-9e15-dca36cce3faf" containerName="maas-api" Jul 24 16:23:08.765723 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.765701 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0aa0f01-f119-49ba-9e15-dca36cce3faf" containerName="maas-api" Jul 24 16:23:08.765853 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.765806 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0aa0f01-f119-49ba-9e15-dca36cce3faf" containerName="maas-api" Jul 24 16:23:08.769170 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.769149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.772029 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.771997 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e12d61-serving-cert\"" Jul 24 16:23:08.778865 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.778841 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c"] Jul 24 16:23:08.882471 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.882432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb9888-5b39-45dc-8e40-a3f0028b4e73-maas-api-tls\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.882662 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.882486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtn5n\" (UniqueName: \"kubernetes.io/projected/9beb9888-5b39-45dc-8e40-a3f0028b4e73-kube-api-access-jtn5n\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.882662 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.882574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9888-5b39-45dc-8e40-a3f0028b4e73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.984117 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.984080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb9888-5b39-45dc-8e40-a3f0028b4e73-maas-api-tls\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.984301 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.984142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtn5n\" (UniqueName: \"kubernetes.io/projected/9beb9888-5b39-45dc-8e40-a3f0028b4e73-kube-api-access-jtn5n\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.984301 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.984205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9888-5b39-45dc-8e40-a3f0028b4e73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.984964 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.984865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9888-5b39-45dc-8e40-a3f0028b4e73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.986705 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.986684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb9888-5b39-45dc-8e40-a3f0028b4e73-maas-api-tls\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:08.992180 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:08.992156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtn5n\" (UniqueName: \"kubernetes.io/projected/9beb9888-5b39-45dc-8e40-a3f0028b4e73-kube-api-access-jtn5n\") pod \"maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:09.082502 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:09.082414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:09.225298 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:09.225269 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c"] Jul 24 16:23:09.228052 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:23:09.228022 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9beb9888_5b39_45dc_8e40_a3f0028b4e73.slice/crio-34746d56b8ea95f0ca986ba69714ecf388620426c3105b6fd05bb7a970fd9359 WatchSource:0}: Error finding container 34746d56b8ea95f0ca986ba69714ecf388620426c3105b6fd05bb7a970fd9359: Status 404 returned error can't find the container with id 34746d56b8ea95f0ca986ba69714ecf388620426c3105b6fd05bb7a970fd9359 Jul 24 16:23:10.113166 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:10.113128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" event={"ID":"9beb9888-5b39-45dc-8e40-a3f0028b4e73","Type":"ContainerStarted","Data":"f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c"} Jul 24 16:23:10.113599 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:10.113180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" event={"ID":"9beb9888-5b39-45dc-8e40-a3f0028b4e73","Type":"ContainerStarted","Data":"34746d56b8ea95f0ca986ba69714ecf388620426c3105b6fd05bb7a970fd9359"} Jul 24 16:23:10.113599 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:10.113389 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:10.130863 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:10.130805 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" podStartSLOduration=1.764094695 podStartE2EDuration="2.130785451s" totalImagesPullingTime="366.690756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:23:10.130261296 +0000 UTC m=+2158.651869837" watchObservedRunningTime="2026-07-24 16:23:10.130785451 +0000 UTC m=+2158.652393993" Jul 24 16:23:16.124092 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.124054 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:23:16.756127 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.756095 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g"] Jul 24 16:23:16.759872 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.759849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.764222 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.764201 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d060b0-serving-cert\"" Jul 24 16:23:16.771212 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.771184 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g"] Jul 24 16:23:16.858532 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.858496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbf0779-593c-406f-baa0-c617fdb34ab2-maas-api-tls\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.858700 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.858624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf0779-593c-406f-baa0-c617fdb34ab2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.858700 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.858658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpcs\" (UniqueName: \"kubernetes.io/projected/3cbf0779-593c-406f-baa0-c617fdb34ab2-kube-api-access-vfpcs\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.960175 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.960134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf0779-593c-406f-baa0-c617fdb34ab2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.960175 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.960180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpcs\" (UniqueName: \"kubernetes.io/projected/3cbf0779-593c-406f-baa0-c617fdb34ab2-kube-api-access-vfpcs\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.960412 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.960219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbf0779-593c-406f-baa0-c617fdb34ab2-maas-api-tls\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.960775 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.960739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf0779-593c-406f-baa0-c617fdb34ab2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.962757 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.962739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbf0779-593c-406f-baa0-c617fdb34ab2-maas-api-tls\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:16.974913 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:16.974892 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpcs\" (UniqueName: \"kubernetes.io/projected/3cbf0779-593c-406f-baa0-c617fdb34ab2-kube-api-access-vfpcs\") pod \"maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:17.071072 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:17.071034 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:17.218069 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:17.218003 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g"] Jul 24 16:23:17.220582 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:23:17.220547 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbf0779_593c_406f_baa0_c617fdb34ab2.slice/crio-d15a93184fd59e613759c5b512876b79ceb29c1139428851d820495a7cfc8b69 WatchSource:0}: Error finding container d15a93184fd59e613759c5b512876b79ceb29c1139428851d820495a7cfc8b69: Status 404 returned error can't find the container with id d15a93184fd59e613759c5b512876b79ceb29c1139428851d820495a7cfc8b69 Jul 24 16:23:18.154374 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:18.154342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" event={"ID":"3cbf0779-593c-406f-baa0-c617fdb34ab2","Type":"ContainerStarted","Data":"15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f"} Jul 24 16:23:18.154560 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:18.154391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" event={"ID":"3cbf0779-593c-406f-baa0-c617fdb34ab2","Type":"ContainerStarted","Data":"d15a93184fd59e613759c5b512876b79ceb29c1139428851d820495a7cfc8b69"} Jul 24 16:23:18.154560 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:18.154427 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:23:18.173385 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:18.173339 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" podStartSLOduration=1.867047141 podStartE2EDuration="2.173319255s" totalImagesPullingTime="306.272114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:23:18.17111524 +0000 UTC m=+2166.692723778" watchObservedRunningTime="2026-07-24 16:23:18.173319255 +0000 UTC m=+2166.694927796" Jul 24 16:23:24.164583 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:23:24.164551 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:25:22.866457 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:22.866421 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c"] Jul 24 16:25:22.867033 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:22.866664 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" podUID="9beb9888-5b39-45dc-8e40-a3f0028b4e73" containerName="maas-api" containerID="cri-o://f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c" gracePeriod=30 Jul 24 16:25:23.314391 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.314366 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:25:23.385896 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.385869 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtn5n\" (UniqueName: \"kubernetes.io/projected/9beb9888-5b39-45dc-8e40-a3f0028b4e73-kube-api-access-jtn5n\") pod \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " Jul 24 16:25:23.385896 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.385903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb9888-5b39-45dc-8e40-a3f0028b4e73-maas-api-tls\") pod \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " Jul 24 16:25:23.386149 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.385986 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9888-5b39-45dc-8e40-a3f0028b4e73-openshift-service-ca\") pod \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\" (UID: \"9beb9888-5b39-45dc-8e40-a3f0028b4e73\") " Jul 24 16:25:23.386419 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.386394 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9beb9888-5b39-45dc-8e40-a3f0028b4e73-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9beb9888-5b39-45dc-8e40-a3f0028b4e73" (UID: "9beb9888-5b39-45dc-8e40-a3f0028b4e73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:25:23.387976 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.387947 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9beb9888-5b39-45dc-8e40-a3f0028b4e73-kube-api-access-jtn5n" (OuterVolumeSpecName: "kube-api-access-jtn5n") pod "9beb9888-5b39-45dc-8e40-a3f0028b4e73" (UID: "9beb9888-5b39-45dc-8e40-a3f0028b4e73"). InnerVolumeSpecName "kube-api-access-jtn5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:25:23.388178 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.388156 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9beb9888-5b39-45dc-8e40-a3f0028b4e73-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9beb9888-5b39-45dc-8e40-a3f0028b4e73" (UID: "9beb9888-5b39-45dc-8e40-a3f0028b4e73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:25:23.487683 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.487652 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9888-5b39-45dc-8e40-a3f0028b4e73-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:25:23.487683 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.487679 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtn5n\" (UniqueName: \"kubernetes.io/projected/9beb9888-5b39-45dc-8e40-a3f0028b4e73-kube-api-access-jtn5n\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:25:23.487683 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.487690 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb9888-5b39-45dc-8e40-a3f0028b4e73-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:25:23.752186 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.752102 2580 generic.go:356] "Generic (PLEG): container finished" podID="9beb9888-5b39-45dc-8e40-a3f0028b4e73" containerID="f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c" exitCode=0 Jul 24 16:25:23.752186 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.752148 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" Jul 24 16:25:23.752186 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.752166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" event={"ID":"9beb9888-5b39-45dc-8e40-a3f0028b4e73","Type":"ContainerDied","Data":"f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c"} Jul 24 16:25:23.752451 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.752218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c" event={"ID":"9beb9888-5b39-45dc-8e40-a3f0028b4e73","Type":"ContainerDied","Data":"34746d56b8ea95f0ca986ba69714ecf388620426c3105b6fd05bb7a970fd9359"} Jul 24 16:25:23.752451 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.752244 2580 scope.go:117] "RemoveContainer" containerID="f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c" Jul 24 16:25:23.762078 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.762060 2580 scope.go:117] "RemoveContainer" containerID="f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c" Jul 24 16:25:23.762336 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:25:23.762314 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c\": container with ID starting with f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c not found: ID does not exist" containerID="f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c" Jul 24 16:25:23.762384 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.762350 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c"} err="failed to get container status \"f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c\": rpc error: code = NotFound desc = could not find container \"f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c\": container with ID starting with f278328b6c3e222d6e9a12d8084fd29a5bc23bf95b45d6ae98251a636c18bc6c not found: ID does not exist" Jul 24 16:25:23.773018 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.772996 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c"] Jul 24 16:25:23.777316 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:23.777296 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e12d61-75fb87b8dd-5d75c"] Jul 24 16:25:24.011152 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:24.011128 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9beb9888-5b39-45dc-8e40-a3f0028b4e73" path="/var/lib/kubelet/pods/9beb9888-5b39-45dc-8e40-a3f0028b4e73/volumes" Jul 24 16:25:44.763302 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:44.763268 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g"] Jul 24 16:25:44.763856 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:44.763507 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" podUID="3cbf0779-593c-406f-baa0-c617fdb34ab2" containerName="maas-api" containerID="cri-o://15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f" gracePeriod=30 Jul 24 16:25:45.008366 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.008344 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:25:45.184207 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.184117 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfpcs\" (UniqueName: \"kubernetes.io/projected/3cbf0779-593c-406f-baa0-c617fdb34ab2-kube-api-access-vfpcs\") pod \"3cbf0779-593c-406f-baa0-c617fdb34ab2\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " Jul 24 16:25:45.184207 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.184177 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf0779-593c-406f-baa0-c617fdb34ab2-openshift-service-ca\") pod \"3cbf0779-593c-406f-baa0-c617fdb34ab2\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " Jul 24 16:25:45.184427 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.184252 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbf0779-593c-406f-baa0-c617fdb34ab2-maas-api-tls\") pod \"3cbf0779-593c-406f-baa0-c617fdb34ab2\" (UID: \"3cbf0779-593c-406f-baa0-c617fdb34ab2\") " Jul 24 16:25:45.184569 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.184540 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbf0779-593c-406f-baa0-c617fdb34ab2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cbf0779-593c-406f-baa0-c617fdb34ab2" (UID: "3cbf0779-593c-406f-baa0-c617fdb34ab2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:25:45.186454 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.186425 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbf0779-593c-406f-baa0-c617fdb34ab2-kube-api-access-vfpcs" (OuterVolumeSpecName: "kube-api-access-vfpcs") pod "3cbf0779-593c-406f-baa0-c617fdb34ab2" (UID: "3cbf0779-593c-406f-baa0-c617fdb34ab2"). InnerVolumeSpecName "kube-api-access-vfpcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:25:45.186547 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.186516 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cbf0779-593c-406f-baa0-c617fdb34ab2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cbf0779-593c-406f-baa0-c617fdb34ab2" (UID: "3cbf0779-593c-406f-baa0-c617fdb34ab2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:25:45.285873 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.285843 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfpcs\" (UniqueName: \"kubernetes.io/projected/3cbf0779-593c-406f-baa0-c617fdb34ab2-kube-api-access-vfpcs\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:25:45.285873 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.285872 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf0779-593c-406f-baa0-c617fdb34ab2-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:25:45.286045 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.285885 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbf0779-593c-406f-baa0-c617fdb34ab2-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:25:45.867095 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.867056 2580 generic.go:356] "Generic (PLEG): container finished" podID="3cbf0779-593c-406f-baa0-c617fdb34ab2" containerID="15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f" exitCode=0 Jul 24 16:25:45.867515 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.867107 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" Jul 24 16:25:45.867515 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.867116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" event={"ID":"3cbf0779-593c-406f-baa0-c617fdb34ab2","Type":"ContainerDied","Data":"15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f"} Jul 24 16:25:45.867515 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.867178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g" event={"ID":"3cbf0779-593c-406f-baa0-c617fdb34ab2","Type":"ContainerDied","Data":"d15a93184fd59e613759c5b512876b79ceb29c1139428851d820495a7cfc8b69"} Jul 24 16:25:45.867515 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.867206 2580 scope.go:117] "RemoveContainer" containerID="15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f" Jul 24 16:25:45.876273 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.876256 2580 scope.go:117] "RemoveContainer" containerID="15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f" Jul 24 16:25:45.876514 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:25:45.876497 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f\": container with ID starting with 15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f not found: ID does not exist" containerID="15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f" Jul 24 16:25:45.876553 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.876523 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f"} err="failed to get container status \"15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f\": rpc error: code = NotFound desc = could not find container \"15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f\": container with ID starting with 15d2f6385282c40ba4501505217a622fe4393ec95b839480198d2cef9cfef66f not found: ID does not exist" Jul 24 16:25:45.887462 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.887438 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g"] Jul 24 16:25:45.891739 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:45.891720 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d060b0-7b957595cd-2fn6g"] Jul 24 16:25:46.010673 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:46.010641 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cbf0779-593c-406f-baa0-c617fdb34ab2" path="/var/lib/kubelet/pods/3cbf0779-593c-406f-baa0-c617fdb34ab2/volumes" Jul 24 16:25:59.781665 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.781627 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm"] Jul 24 16:25:59.782320 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.782022 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9beb9888-5b39-45dc-8e40-a3f0028b4e73" containerName="maas-api" Jul 24 16:25:59.782320 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.782034 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9beb9888-5b39-45dc-8e40-a3f0028b4e73" containerName="maas-api" Jul 24 16:25:59.782320 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.782055 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbf0779-593c-406f-baa0-c617fdb34ab2" containerName="maas-api" Jul 24 16:25:59.782320 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.782061 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbf0779-593c-406f-baa0-c617fdb34ab2" containerName="maas-api" Jul 24 16:25:59.782320 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.782120 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9beb9888-5b39-45dc-8e40-a3f0028b4e73" containerName="maas-api" Jul 24 16:25:59.782320 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.782130 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbf0779-593c-406f-baa0-c617fdb34ab2" containerName="maas-api" Jul 24 16:25:59.785368 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.785346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.787568 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.787542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-01d58c-serving-cert\"" Jul 24 16:25:59.795681 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.795659 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm"] Jul 24 16:25:59.814268 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.814237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.814428 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.814276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5tn\" (UniqueName: \"kubernetes.io/projected/6a1ad33a-efe6-4793-8f48-6df947a0de51-kube-api-access-bv5tn\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.814428 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.814327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ad33a-efe6-4793-8f48-6df947a0de51-openshift-service-ca\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.915750 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.915712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ad33a-efe6-4793-8f48-6df947a0de51-openshift-service-ca\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.915971 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.915798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.915971 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.915822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5tn\" (UniqueName: \"kubernetes.io/projected/6a1ad33a-efe6-4793-8f48-6df947a0de51-kube-api-access-bv5tn\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.916072 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:25:59.915986 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-serving-cert: secret "maas-api-e2e-api-a-01d58c-serving-cert" not found Jul 24 16:25:59.916072 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:25:59.916065 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls podName:6a1ad33a-efe6-4793-8f48-6df947a0de51 nodeName:}" failed. No retries permitted until 2026-07-24 16:26:00.416042832 +0000 UTC m=+2328.937651350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls") pod "maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" (UID: "6a1ad33a-efe6-4793-8f48-6df947a0de51") : secret "maas-api-e2e-api-a-01d58c-serving-cert" not found Jul 24 16:25:59.916552 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.916534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ad33a-efe6-4793-8f48-6df947a0de51-openshift-service-ca\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:25:59.925156 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:25:59.925135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5tn\" (UniqueName: \"kubernetes.io/projected/6a1ad33a-efe6-4793-8f48-6df947a0de51-kube-api-access-bv5tn\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:00.421395 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:00.421361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:00.424274 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:00.424246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls\") pod \"maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:00.696305 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:00.696206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:00.843415 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:00.843380 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm"] Jul 24 16:26:00.847152 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:26:00.847123 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1ad33a_efe6_4793_8f48_6df947a0de51.slice/crio-2bb11d5fed228a3220e44a5c0e2d46ebc9796e0227fe23d4663eee94aefb8728 WatchSource:0}: Error finding container 2bb11d5fed228a3220e44a5c0e2d46ebc9796e0227fe23d4663eee94aefb8728: Status 404 returned error can't find the container with id 2bb11d5fed228a3220e44a5c0e2d46ebc9796e0227fe23d4663eee94aefb8728 Jul 24 16:26:00.848531 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:00.848509 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:26:00.941555 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:00.941517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" event={"ID":"6a1ad33a-efe6-4793-8f48-6df947a0de51","Type":"ContainerStarted","Data":"2bb11d5fed228a3220e44a5c0e2d46ebc9796e0227fe23d4663eee94aefb8728"} Jul 24 16:26:01.948547 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:01.948513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" event={"ID":"6a1ad33a-efe6-4793-8f48-6df947a0de51","Type":"ContainerStarted","Data":"da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36"} Jul 24 16:26:01.948953 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:01.948586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:01.970269 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:01.970211 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" podStartSLOduration=2.481772671 podStartE2EDuration="2.970190528s" totalImagesPullingTime="488.417857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:26:01.967981101 +0000 UTC m=+2330.489589638" watchObservedRunningTime="2026-07-24 16:26:01.970190528 +0000 UTC m=+2330.491799068" Jul 24 16:26:07.748208 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.748121 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2"] Jul 24 16:26:07.751968 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.751927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.754896 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.754877 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2c74ca-serving-cert\"" Jul 24 16:26:07.767022 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.766999 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2"] Jul 24 16:26:07.789113 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.789071 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2503e5b0-c827-4180-abd3-9d82891d6283-openshift-service-ca\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.789298 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.789133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2rr\" (UniqueName: \"kubernetes.io/projected/2503e5b0-c827-4180-abd3-9d82891d6283-kube-api-access-hg2rr\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.789298 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.789234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.890104 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.890069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.890285 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.890125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2503e5b0-c827-4180-abd3-9d82891d6283-openshift-service-ca\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.890285 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.890172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2rr\" (UniqueName: \"kubernetes.io/projected/2503e5b0-c827-4180-abd3-9d82891d6283-kube-api-access-hg2rr\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.890285 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:26:07.890217 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-serving-cert: secret "maas-api-e2e-api-b-2c74ca-serving-cert" not found Jul 24 16:26:07.890390 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:26:07.890295 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls podName:2503e5b0-c827-4180-abd3-9d82891d6283 nodeName:}" failed. No retries permitted until 2026-07-24 16:26:08.390274774 +0000 UTC m=+2336.911883296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls") pod "maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" (UID: "2503e5b0-c827-4180-abd3-9d82891d6283") : secret "maas-api-e2e-api-b-2c74ca-serving-cert" not found Jul 24 16:26:07.890773 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.890756 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2503e5b0-c827-4180-abd3-9d82891d6283-openshift-service-ca\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.902016 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.901989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2rr\" (UniqueName: \"kubernetes.io/projected/2503e5b0-c827-4180-abd3-9d82891d6283-kube-api-access-hg2rr\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:07.959156 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:07.959128 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:08.394497 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:08.394462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:08.396965 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:08.396918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls\") pod \"maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:08.661986 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:08.661886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:08.813151 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:08.813116 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2"] Jul 24 16:26:08.816281 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:26:08.816248 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2503e5b0_c827_4180_abd3_9d82891d6283.slice/crio-860f3f898e912665b6d3c92aacceac3d385561ce719ef4775fa1290595ef0f74 WatchSource:0}: Error finding container 860f3f898e912665b6d3c92aacceac3d385561ce719ef4775fa1290595ef0f74: Status 404 returned error can't find the container with id 860f3f898e912665b6d3c92aacceac3d385561ce719ef4775fa1290595ef0f74 Jul 24 16:26:08.989765 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:08.989666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" event={"ID":"2503e5b0-c827-4180-abd3-9d82891d6283","Type":"ContainerStarted","Data":"860f3f898e912665b6d3c92aacceac3d385561ce719ef4775fa1290595ef0f74"} Jul 24 16:26:09.995772 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:09.995730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" event={"ID":"2503e5b0-c827-4180-abd3-9d82891d6283","Type":"ContainerStarted","Data":"4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664"} Jul 24 16:26:09.996205 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:09.995850 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:10.011675 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:10.011633 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" podStartSLOduration=2.659407415 podStartE2EDuration="3.011618749s" totalImagesPullingTime="352.211334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:26:10.010633483 +0000 UTC m=+2338.532242049" watchObservedRunningTime="2026-07-24 16:26:10.011618749 +0000 UTC m=+2338.533227288" Jul 24 16:26:16.010759 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:16.010734 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:26:36.035841 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.035808 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm"] Jul 24 16:26:36.036315 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.036095 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" podUID="6a1ad33a-efe6-4793-8f48-6df947a0de51" containerName="maas-api" containerID="cri-o://da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36" gracePeriod=30 Jul 24 16:26:36.290780 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.290718 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:36.356367 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.356331 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv5tn\" (UniqueName: \"kubernetes.io/projected/6a1ad33a-efe6-4793-8f48-6df947a0de51-kube-api-access-bv5tn\") pod \"6a1ad33a-efe6-4793-8f48-6df947a0de51\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " Jul 24 16:26:36.356542 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.356433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls\") pod \"6a1ad33a-efe6-4793-8f48-6df947a0de51\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " Jul 24 16:26:36.356542 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.356466 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ad33a-efe6-4793-8f48-6df947a0de51-openshift-service-ca\") pod \"6a1ad33a-efe6-4793-8f48-6df947a0de51\" (UID: \"6a1ad33a-efe6-4793-8f48-6df947a0de51\") " Jul 24 16:26:36.356871 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.356850 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1ad33a-efe6-4793-8f48-6df947a0de51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a1ad33a-efe6-4793-8f48-6df947a0de51" (UID: "6a1ad33a-efe6-4793-8f48-6df947a0de51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:26:36.358581 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.358559 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a1ad33a-efe6-4793-8f48-6df947a0de51" (UID: "6a1ad33a-efe6-4793-8f48-6df947a0de51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:26:36.358581 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.358575 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1ad33a-efe6-4793-8f48-6df947a0de51-kube-api-access-bv5tn" (OuterVolumeSpecName: "kube-api-access-bv5tn") pod "6a1ad33a-efe6-4793-8f48-6df947a0de51" (UID: "6a1ad33a-efe6-4793-8f48-6df947a0de51"). InnerVolumeSpecName "kube-api-access-bv5tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:26:36.457293 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.457258 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1ad33a-efe6-4793-8f48-6df947a0de51-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:26:36.457293 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.457289 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1ad33a-efe6-4793-8f48-6df947a0de51-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:26:36.457293 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:36.457300 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv5tn\" (UniqueName: \"kubernetes.io/projected/6a1ad33a-efe6-4793-8f48-6df947a0de51-kube-api-access-bv5tn\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:26:37.126056 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.126018 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a1ad33a-efe6-4793-8f48-6df947a0de51" containerID="da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36" exitCode=0 Jul 24 16:26:37.126530 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.126064 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" Jul 24 16:26:37.126530 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.126111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" event={"ID":"6a1ad33a-efe6-4793-8f48-6df947a0de51","Type":"ContainerDied","Data":"da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36"} Jul 24 16:26:37.126530 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.126179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm" event={"ID":"6a1ad33a-efe6-4793-8f48-6df947a0de51","Type":"ContainerDied","Data":"2bb11d5fed228a3220e44a5c0e2d46ebc9796e0227fe23d4663eee94aefb8728"} Jul 24 16:26:37.126530 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.126214 2580 scope.go:117] "RemoveContainer" containerID="da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36" Jul 24 16:26:37.135584 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.135566 2580 scope.go:117] "RemoveContainer" containerID="da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36" Jul 24 16:26:37.135861 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:26:37.135840 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36\": container with ID starting with da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36 not found: ID does not exist" containerID="da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36" Jul 24 16:26:37.135919 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.135870 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36"} err="failed to get container status \"da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36\": rpc error: code = NotFound desc = could not find container \"da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36\": container with ID starting with da909ccb175796d420958087f79207a680b841e3a61976683d7b50cc67bc4a36 not found: ID does not exist" Jul 24 16:26:37.147240 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.147218 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm"] Jul 24 16:26:37.151645 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:37.151621 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01d58c-db8bf47cc-slnsm"] Jul 24 16:26:38.010265 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:26:38.010237 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1ad33a-efe6-4793-8f48-6df947a0de51" path="/var/lib/kubelet/pods/6a1ad33a-efe6-4793-8f48-6df947a0de51/volumes" Jul 24 16:27:02.904213 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:02.904176 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2"] Jul 24 16:27:02.904741 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:02.904478 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" podUID="2503e5b0-c827-4180-abd3-9d82891d6283" containerName="maas-api" containerID="cri-o://4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664" gracePeriod=30 Jul 24 16:27:03.156665 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.156605 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:27:03.257420 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.257383 2580 generic.go:356] "Generic (PLEG): container finished" podID="2503e5b0-c827-4180-abd3-9d82891d6283" containerID="4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664" exitCode=0 Jul 24 16:27:03.257420 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.257415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" event={"ID":"2503e5b0-c827-4180-abd3-9d82891d6283","Type":"ContainerDied","Data":"4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664"} Jul 24 16:27:03.257673 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.257438 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" Jul 24 16:27:03.257673 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.257470 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2" event={"ID":"2503e5b0-c827-4180-abd3-9d82891d6283","Type":"ContainerDied","Data":"860f3f898e912665b6d3c92aacceac3d385561ce719ef4775fa1290595ef0f74"} Jul 24 16:27:03.257673 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.257494 2580 scope.go:117] "RemoveContainer" containerID="4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664" Jul 24 16:27:03.266476 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.266454 2580 scope.go:117] "RemoveContainer" containerID="4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664" Jul 24 16:27:03.266775 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:27:03.266757 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664\": container with ID starting with 4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664 not found: ID does not exist" containerID="4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664" Jul 24 16:27:03.266827 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.266791 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664"} err="failed to get container status \"4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664\": rpc error: code = NotFound desc = could not find container \"4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664\": container with ID starting with 4a844a91db5401d7da677a22d6e915365079eb6ad6bf9d432d585556fe38a664 not found: ID does not exist" Jul 24 16:27:03.297144 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.297118 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2503e5b0-c827-4180-abd3-9d82891d6283-openshift-service-ca\") pod \"2503e5b0-c827-4180-abd3-9d82891d6283\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " Jul 24 16:27:03.297281 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.297155 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg2rr\" (UniqueName: \"kubernetes.io/projected/2503e5b0-c827-4180-abd3-9d82891d6283-kube-api-access-hg2rr\") pod \"2503e5b0-c827-4180-abd3-9d82891d6283\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " Jul 24 16:27:03.297281 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.297181 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls\") pod \"2503e5b0-c827-4180-abd3-9d82891d6283\" (UID: \"2503e5b0-c827-4180-abd3-9d82891d6283\") " Jul 24 16:27:03.297523 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.297500 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2503e5b0-c827-4180-abd3-9d82891d6283-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2503e5b0-c827-4180-abd3-9d82891d6283" (UID: "2503e5b0-c827-4180-abd3-9d82891d6283"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:27:03.299344 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.299321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2503e5b0-c827-4180-abd3-9d82891d6283-kube-api-access-hg2rr" (OuterVolumeSpecName: "kube-api-access-hg2rr") pod "2503e5b0-c827-4180-abd3-9d82891d6283" (UID: "2503e5b0-c827-4180-abd3-9d82891d6283"). InnerVolumeSpecName "kube-api-access-hg2rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:27:03.299344 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.299330 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2503e5b0-c827-4180-abd3-9d82891d6283" (UID: "2503e5b0-c827-4180-abd3-9d82891d6283"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:27:03.398094 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.398066 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2503e5b0-c827-4180-abd3-9d82891d6283-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:27:03.398094 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.398087 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2rr\" (UniqueName: \"kubernetes.io/projected/2503e5b0-c827-4180-abd3-9d82891d6283-kube-api-access-hg2rr\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:27:03.398094 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.398099 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2503e5b0-c827-4180-abd3-9d82891d6283-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:27:03.578753 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.578724 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2"] Jul 24 16:27:03.582048 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:03.582022 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c74ca-fc8c44767-4qhc2"] Jul 24 16:27:04.010419 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:04.010391 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2503e5b0-c827-4180-abd3-9d82891d6283" path="/var/lib/kubelet/pods/2503e5b0-c827-4180-abd3-9d82891d6283/volumes" Jul 24 16:27:12.124036 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:12.124005 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:27:12.131129 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:12.131107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:27:18.187270 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.187236 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm"] Jul 24 16:27:18.187679 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.187666 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2503e5b0-c827-4180-abd3-9d82891d6283" containerName="maas-api" Jul 24 16:27:18.187737 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.187681 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2503e5b0-c827-4180-abd3-9d82891d6283" containerName="maas-api" Jul 24 16:27:18.187737 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.187692 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1ad33a-efe6-4793-8f48-6df947a0de51" containerName="maas-api" Jul 24 16:27:18.187737 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.187699 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1ad33a-efe6-4793-8f48-6df947a0de51" containerName="maas-api" Jul 24 16:27:18.187835 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.187787 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2503e5b0-c827-4180-abd3-9d82891d6283" containerName="maas-api" Jul 24 16:27:18.187835 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.187797 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1ad33a-efe6-4793-8f48-6df947a0de51" containerName="maas-api" Jul 24 16:27:18.190922 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.190905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.193258 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.193235 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a5fa08-serving-cert\"" Jul 24 16:27:18.199498 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.199469 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm"] Jul 24 16:27:18.332793 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.332759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb97dad9-0828-41da-a20b-633d509f61b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.332986 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.332837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5s4\" (UniqueName: \"kubernetes.io/projected/cb97dad9-0828-41da-a20b-633d509f61b7-kube-api-access-jp5s4\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.332986 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.332859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb97dad9-0828-41da-a20b-633d509f61b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.434049 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.434018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5s4\" (UniqueName: \"kubernetes.io/projected/cb97dad9-0828-41da-a20b-633d509f61b7-kube-api-access-jp5s4\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.434190 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.434060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb97dad9-0828-41da-a20b-633d509f61b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.434190 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.434136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb97dad9-0828-41da-a20b-633d509f61b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.434792 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.434775 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb97dad9-0828-41da-a20b-633d509f61b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.436442 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.436424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb97dad9-0828-41da-a20b-633d509f61b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.442612 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.442566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5s4\" (UniqueName: \"kubernetes.io/projected/cb97dad9-0828-41da-a20b-633d509f61b7-kube-api-access-jp5s4\") pod \"maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.502145 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.502115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:18.642775 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:18.642647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm"] Jul 24 16:27:18.645540 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:27:18.645513 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb97dad9_0828_41da_a20b_633d509f61b7.slice/crio-272b4cf6b87a78505049a6eff040fbbe2a290f666e791dcf89089993e007c74f WatchSource:0}: Error finding container 272b4cf6b87a78505049a6eff040fbbe2a290f666e791dcf89089993e007c74f: Status 404 returned error can't find the container with id 272b4cf6b87a78505049a6eff040fbbe2a290f666e791dcf89089993e007c74f Jul 24 16:27:19.337854 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:19.337812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" event={"ID":"cb97dad9-0828-41da-a20b-633d509f61b7","Type":"ContainerStarted","Data":"6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976"} Jul 24 16:27:19.338344 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:19.337898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" event={"ID":"cb97dad9-0828-41da-a20b-633d509f61b7","Type":"ContainerStarted","Data":"272b4cf6b87a78505049a6eff040fbbe2a290f666e791dcf89089993e007c74f"} Jul 24 16:27:19.338344 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:19.337961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:19.355475 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:19.355377 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" podStartSLOduration=0.898467016 podStartE2EDuration="1.355358335s" totalImagesPullingTime="456.891319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:27:19.353750781 +0000 UTC m=+2407.875359320" watchObservedRunningTime="2026-07-24 16:27:19.355358335 +0000 UTC m=+2407.876966875" Jul 24 16:27:25.348321 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:25.348295 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:27:26.131687 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.131655 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96"] Jul 24 16:27:26.135173 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.135149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.137369 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.137345 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-675edf-serving-cert\"" Jul 24 16:27:26.144926 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.144897 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96"] Jul 24 16:27:26.204166 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.204131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-openshift-service-ca\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.204166 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.204168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.204401 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.204204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4wr\" (UniqueName: \"kubernetes.io/projected/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-kube-api-access-tl4wr\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.305160 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.305127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-openshift-service-ca\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.305308 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.305167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.305308 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.305195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4wr\" (UniqueName: \"kubernetes.io/projected/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-kube-api-access-tl4wr\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.305308 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:27:26.305281 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-serving-cert: secret "maas-api-e2e-shared-b-675edf-serving-cert" not found Jul 24 16:27:26.305439 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:27:26.305343 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls podName:92bc2101-6c00-43b7-99e0-1ff6cfbbf250 nodeName:}" failed. No retries permitted until 2026-07-24 16:27:26.805325798 +0000 UTC m=+2415.326934317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls") pod "maas-api-e2e-shared-b-675edf-57757648fc-64d96" (UID: "92bc2101-6c00-43b7-99e0-1ff6cfbbf250") : secret "maas-api-e2e-shared-b-675edf-serving-cert" not found Jul 24 16:27:26.305725 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.305706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-openshift-service-ca\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.317462 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.317437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4wr\" (UniqueName: \"kubernetes.io/projected/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-kube-api-access-tl4wr\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.809200 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.809170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:26.811667 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:26.811645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls\") pod \"maas-api-e2e-shared-b-675edf-57757648fc-64d96\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:27.048188 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:27.048151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:27.186129 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:27.186102 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96"] Jul 24 16:27:27.188295 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:27:27.188265 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bc2101_6c00_43b7_99e0_1ff6cfbbf250.slice/crio-2aa7d9b992fb2423948b0b8a613ff8ab3e452ce3b5e1233983b445acb745cc8a WatchSource:0}: Error finding container 2aa7d9b992fb2423948b0b8a613ff8ab3e452ce3b5e1233983b445acb745cc8a: Status 404 returned error can't find the container with id 2aa7d9b992fb2423948b0b8a613ff8ab3e452ce3b5e1233983b445acb745cc8a Jul 24 16:27:27.376844 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:27.376753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" event={"ID":"92bc2101-6c00-43b7-99e0-1ff6cfbbf250","Type":"ContainerStarted","Data":"2aa7d9b992fb2423948b0b8a613ff8ab3e452ce3b5e1233983b445acb745cc8a"} Jul 24 16:27:28.382903 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:28.382863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" event={"ID":"92bc2101-6c00-43b7-99e0-1ff6cfbbf250","Type":"ContainerStarted","Data":"350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7"} Jul 24 16:27:28.383436 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:28.383132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:27:28.405194 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:28.405148 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" podStartSLOduration=2.07794979 podStartE2EDuration="2.40513368s" totalImagesPullingTime="327.18389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:27:28.403128088 +0000 UTC m=+2416.924736628" watchObservedRunningTime="2026-07-24 16:27:28.40513368 +0000 UTC m=+2416.926742220" Jul 24 16:27:34.393818 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:27:34.393735 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:32:12.155343 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:32:12.155263 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:32:12.163277 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:32:12.163259 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:36:39.565852 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.565775 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7407-140271939632560"] Jul 24 16:36:39.571048 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.570257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:39.573402 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.573373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kxnzx\"" Jul 24 16:36:39.576749 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.576722 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939632560"] Jul 24 16:36:39.634413 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.634392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdm8\" (UniqueName: \"kubernetes.io/projected/3b32ea03-7790-4305-aba9-af8b4b3c072e-kube-api-access-qhdm8\") pod \"test-curl-7407-140271939632560\" (UID: \"3b32ea03-7790-4305-aba9-af8b4b3c072e\") " pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:39.735193 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.735158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdm8\" (UniqueName: \"kubernetes.io/projected/3b32ea03-7790-4305-aba9-af8b4b3c072e-kube-api-access-qhdm8\") pod \"test-curl-7407-140271939632560\" (UID: \"3b32ea03-7790-4305-aba9-af8b4b3c072e\") " pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:39.742961 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.742918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdm8\" (UniqueName: \"kubernetes.io/projected/3b32ea03-7790-4305-aba9-af8b4b3c072e-kube-api-access-qhdm8\") pod \"test-curl-7407-140271939632560\" (UID: \"3b32ea03-7790-4305-aba9-af8b4b3c072e\") " pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:39.882997 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:39.882906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:40.013803 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:40.013777 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939632560"] Jul 24 16:36:40.015771 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:36:40.015740 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b32ea03_7790_4305_aba9_af8b4b3c072e.slice/crio-cdec952b5225d2a6a714c05451bbff91e3c7b80fdb029db1fd319b643f0ed775 WatchSource:0}: Error finding container cdec952b5225d2a6a714c05451bbff91e3c7b80fdb029db1fd319b643f0ed775: Status 404 returned error can't find the container with id cdec952b5225d2a6a714c05451bbff91e3c7b80fdb029db1fd319b643f0ed775 Jul 24 16:36:40.017170 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:40.017146 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:36:40.999777 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:40.999703 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b32ea03-7790-4305-aba9-af8b4b3c072e" containerID="81a1e9e5f6e48e9545efe15af7e0599d9fb3534a330133aa93b6555ddd155128" exitCode=0 Jul 24 16:36:40.999777 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:40.999758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939632560" event={"ID":"3b32ea03-7790-4305-aba9-af8b4b3c072e","Type":"ContainerDied","Data":"81a1e9e5f6e48e9545efe15af7e0599d9fb3534a330133aa93b6555ddd155128"} Jul 24 16:36:41.000244 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:40.999783 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939632560" event={"ID":"3b32ea03-7790-4305-aba9-af8b4b3c072e","Type":"ContainerStarted","Data":"cdec952b5225d2a6a714c05451bbff91e3c7b80fdb029db1fd319b643f0ed775"} Jul 24 16:36:42.143315 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.143293 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:42.174524 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.174499 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7407-140271939632560_3b32ea03-7790-4305-aba9-af8b4b3c072e/test-curl-7407-140271939632560/0.log" Jul 24 16:36:42.197291 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.197266 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7407-140271939632560"] Jul 24 16:36:42.200210 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.200185 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7407-140271939632560"] Jul 24 16:36:42.259607 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.259534 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhdm8\" (UniqueName: \"kubernetes.io/projected/3b32ea03-7790-4305-aba9-af8b4b3c072e-kube-api-access-qhdm8\") pod \"3b32ea03-7790-4305-aba9-af8b4b3c072e\" (UID: \"3b32ea03-7790-4305-aba9-af8b4b3c072e\") " Jul 24 16:36:42.261703 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.261679 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b32ea03-7790-4305-aba9-af8b4b3c072e-kube-api-access-qhdm8" (OuterVolumeSpecName: "kube-api-access-qhdm8") pod "3b32ea03-7790-4305-aba9-af8b4b3c072e" (UID: "3b32ea03-7790-4305-aba9-af8b4b3c072e"). InnerVolumeSpecName "kube-api-access-qhdm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:36:42.317913 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.317886 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7407-140271939784880"] Jul 24 16:36:42.318386 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.318370 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b32ea03-7790-4305-aba9-af8b4b3c072e" containerName="test-curl-7407-140271939632560" Jul 24 16:36:42.318440 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.318389 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b32ea03-7790-4305-aba9-af8b4b3c072e" containerName="test-curl-7407-140271939632560" Jul 24 16:36:42.318479 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.318456 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b32ea03-7790-4305-aba9-af8b4b3c072e" containerName="test-curl-7407-140271939632560" Jul 24 16:36:42.321491 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.321477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939784880" Jul 24 16:36:42.326530 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.326503 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939784880"] Jul 24 16:36:42.360402 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.360372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2sh\" (UniqueName: \"kubernetes.io/projected/72901278-f1eb-448a-98db-bd40be6f98b3-kube-api-access-ns2sh\") pod \"test-curl-7407-140271939784880\" (UID: \"72901278-f1eb-448a-98db-bd40be6f98b3\") " pod="opendatahub/test-curl-7407-140271939784880" Jul 24 16:36:42.360566 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.360544 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdm8\" (UniqueName: \"kubernetes.io/projected/3b32ea03-7790-4305-aba9-af8b4b3c072e-kube-api-access-qhdm8\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:36:42.461798 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.461766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2sh\" (UniqueName: \"kubernetes.io/projected/72901278-f1eb-448a-98db-bd40be6f98b3-kube-api-access-ns2sh\") pod \"test-curl-7407-140271939784880\" (UID: \"72901278-f1eb-448a-98db-bd40be6f98b3\") " pod="opendatahub/test-curl-7407-140271939784880" Jul 24 16:36:42.470091 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.470061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2sh\" (UniqueName: \"kubernetes.io/projected/72901278-f1eb-448a-98db-bd40be6f98b3-kube-api-access-ns2sh\") pod \"test-curl-7407-140271939784880\" (UID: \"72901278-f1eb-448a-98db-bd40be6f98b3\") " pod="opendatahub/test-curl-7407-140271939784880" Jul 24 16:36:42.631694 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.631615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939784880" Jul 24 16:36:42.764032 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:42.764008 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939784880"] Jul 24 16:36:42.766070 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:36:42.766039 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72901278_f1eb_448a_98db_bd40be6f98b3.slice/crio-b8690680f9d7262290b32dd45d956ccc3e81b79cea8ed7e76e974f03e0974f1f WatchSource:0}: Error finding container b8690680f9d7262290b32dd45d956ccc3e81b79cea8ed7e76e974f03e0974f1f: Status 404 returned error can't find the container with id b8690680f9d7262290b32dd45d956ccc3e81b79cea8ed7e76e974f03e0974f1f Jul 24 16:36:43.011389 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:43.011366 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:43.011548 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:43.011399 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdec952b5225d2a6a714c05451bbff91e3c7b80fdb029db1fd319b643f0ed775" Jul 24 16:36:43.012519 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:43.012494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939784880" event={"ID":"72901278-f1eb-448a-98db-bd40be6f98b3","Type":"ContainerStarted","Data":"b8690680f9d7262290b32dd45d956ccc3e81b79cea8ed7e76e974f03e0974f1f"} Jul 24 16:36:43.021733 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:36:43.021706 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7407-140271939632560\" is forbidden: User \"system:node:ip-10-0-140-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-155.ec2.internal' and this object" podUID="3b32ea03-7790-4305-aba9-af8b4b3c072e" pod="opendatahub/test-curl-7407-140271939632560" Jul 24 16:36:44.010978 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:44.010924 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b32ea03-7790-4305-aba9-af8b4b3c072e" path="/var/lib/kubelet/pods/3b32ea03-7790-4305-aba9-af8b4b3c072e/volumes" Jul 24 16:36:44.018656 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:44.018626 2580 generic.go:356] "Generic (PLEG): container finished" podID="72901278-f1eb-448a-98db-bd40be6f98b3" containerID="30cc4094cf80eb5a29a053291c650f87fb01e9cedeecd5f032eeb5e9af058172" exitCode=0 Jul 24 16:36:44.018797 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:44.018692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939784880" event={"ID":"72901278-f1eb-448a-98db-bd40be6f98b3","Type":"ContainerDied","Data":"30cc4094cf80eb5a29a053291c650f87fb01e9cedeecd5f032eeb5e9af058172"} Jul 24 16:36:45.152580 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.152558 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939784880" Jul 24 16:36:45.170825 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.170803 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7407-140271939784880_72901278-f1eb-448a-98db-bd40be6f98b3/test-curl-7407-140271939784880/0.log" Jul 24 16:36:45.187741 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.187716 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns2sh\" (UniqueName: \"kubernetes.io/projected/72901278-f1eb-448a-98db-bd40be6f98b3-kube-api-access-ns2sh\") pod \"72901278-f1eb-448a-98db-bd40be6f98b3\" (UID: \"72901278-f1eb-448a-98db-bd40be6f98b3\") " Jul 24 16:36:45.189892 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.189854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72901278-f1eb-448a-98db-bd40be6f98b3-kube-api-access-ns2sh" (OuterVolumeSpecName: "kube-api-access-ns2sh") pod "72901278-f1eb-448a-98db-bd40be6f98b3" (UID: "72901278-f1eb-448a-98db-bd40be6f98b3"). InnerVolumeSpecName "kube-api-access-ns2sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:36:45.192351 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.192327 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7407-140271939784880"] Jul 24 16:36:45.195929 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.195906 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7407-140271939784880"] Jul 24 16:36:45.289137 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.289050 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2sh\" (UniqueName: \"kubernetes.io/projected/72901278-f1eb-448a-98db-bd40be6f98b3-kube-api-access-ns2sh\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:36:45.347716 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.347676 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:45.348113 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.348099 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72901278-f1eb-448a-98db-bd40be6f98b3" containerName="test-curl-7407-140271939784880" Jul 24 16:36:45.348168 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.348115 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72901278-f1eb-448a-98db-bd40be6f98b3" containerName="test-curl-7407-140271939784880" Jul 24 16:36:45.348208 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.348174 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72901278-f1eb-448a-98db-bd40be6f98b3" containerName="test-curl-7407-140271939784880" Jul 24 16:36:45.351151 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.351136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:45.356771 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.356747 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:45.389976 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.389930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrc6\" (UniqueName: \"kubernetes.io/projected/c64e1148-79c9-4b6c-9cfb-d139e10e4222-kube-api-access-9mrc6\") pod \"test-curl-7407-140271940103312\" (UID: \"c64e1148-79c9-4b6c-9cfb-d139e10e4222\") " pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:45.490665 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.490636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrc6\" (UniqueName: \"kubernetes.io/projected/c64e1148-79c9-4b6c-9cfb-d139e10e4222-kube-api-access-9mrc6\") pod \"test-curl-7407-140271940103312\" (UID: \"c64e1148-79c9-4b6c-9cfb-d139e10e4222\") " pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:45.498039 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.498011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrc6\" (UniqueName: \"kubernetes.io/projected/c64e1148-79c9-4b6c-9cfb-d139e10e4222-kube-api-access-9mrc6\") pod \"test-curl-7407-140271940103312\" (UID: \"c64e1148-79c9-4b6c-9cfb-d139e10e4222\") " pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:45.661661 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.661581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:45.795166 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:45.795140 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:45.797804 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:36:45.797774 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc64e1148_79c9_4b6c_9cfb_d139e10e4222.slice/crio-5390f967d2659b87715bb7a4d72fe2c9d4254b5a29b6829c39095a92f17eab8e WatchSource:0}: Error finding container 5390f967d2659b87715bb7a4d72fe2c9d4254b5a29b6829c39095a92f17eab8e: Status 404 returned error can't find the container with id 5390f967d2659b87715bb7a4d72fe2c9d4254b5a29b6829c39095a92f17eab8e Jul 24 16:36:46.011227 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:46.011195 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72901278-f1eb-448a-98db-bd40be6f98b3" path="/var/lib/kubelet/pods/72901278-f1eb-448a-98db-bd40be6f98b3/volumes" Jul 24 16:36:46.028490 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:46.028470 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939784880" Jul 24 16:36:46.028609 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:46.028511 2580 scope.go:117] "RemoveContainer" containerID="30cc4094cf80eb5a29a053291c650f87fb01e9cedeecd5f032eeb5e9af058172" Jul 24 16:36:46.029797 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:46.029769 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271940103312" event={"ID":"c64e1148-79c9-4b6c-9cfb-d139e10e4222","Type":"ContainerStarted","Data":"5390f967d2659b87715bb7a4d72fe2c9d4254b5a29b6829c39095a92f17eab8e"} Jul 24 16:36:47.035491 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:47.035457 2580 generic.go:356] "Generic (PLEG): container finished" podID="c64e1148-79c9-4b6c-9cfb-d139e10e4222" containerID="4ff2a1c9628b88b6835ebcffe8d176fcf35f2cfed42c43cdae28da7056190636" exitCode=0 Jul 24 16:36:47.035988 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:47.035540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271940103312" event={"ID":"c64e1148-79c9-4b6c-9cfb-d139e10e4222","Type":"ContainerDied","Data":"4ff2a1c9628b88b6835ebcffe8d176fcf35f2cfed42c43cdae28da7056190636"} Jul 24 16:36:48.176049 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.176026 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:48.195057 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.195035 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7407-140271940103312_c64e1148-79c9-4b6c-9cfb-d139e10e4222/test-curl-7407-140271940103312/0.log" Jul 24 16:36:48.215925 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.215902 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mrc6\" (UniqueName: \"kubernetes.io/projected/c64e1148-79c9-4b6c-9cfb-d139e10e4222-kube-api-access-9mrc6\") pod \"c64e1148-79c9-4b6c-9cfb-d139e10e4222\" (UID: \"c64e1148-79c9-4b6c-9cfb-d139e10e4222\") " Jul 24 16:36:48.218246 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.218216 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64e1148-79c9-4b6c-9cfb-d139e10e4222-kube-api-access-9mrc6" (OuterVolumeSpecName: "kube-api-access-9mrc6") pod "c64e1148-79c9-4b6c-9cfb-d139e10e4222" (UID: "c64e1148-79c9-4b6c-9cfb-d139e10e4222"). InnerVolumeSpecName "kube-api-access-9mrc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:36:48.219498 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.219479 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:48.223160 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.223139 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:48.316592 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.316534 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mrc6\" (UniqueName: \"kubernetes.io/projected/c64e1148-79c9-4b6c-9cfb-d139e10e4222-kube-api-access-9mrc6\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:36:48.345260 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.345239 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:48.345628 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.345617 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64e1148-79c9-4b6c-9cfb-d139e10e4222" containerName="test-curl-7407-140271940103312" Jul 24 16:36:48.345671 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.345630 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64e1148-79c9-4b6c-9cfb-d139e10e4222" containerName="test-curl-7407-140271940103312" Jul 24 16:36:48.345706 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.345692 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64e1148-79c9-4b6c-9cfb-d139e10e4222" containerName="test-curl-7407-140271940103312" Jul 24 16:36:48.348506 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.348493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:48.352797 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.352775 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:48.417279 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.417251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95p8b\" (UniqueName: \"kubernetes.io/projected/6a45fe8b-da9f-4447-a749-f1808a0ae81c-kube-api-access-95p8b\") pod \"test-curl-7407-140271940103312\" (UID: \"6a45fe8b-da9f-4447-a749-f1808a0ae81c\") " pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:48.518671 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.518647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95p8b\" (UniqueName: \"kubernetes.io/projected/6a45fe8b-da9f-4447-a749-f1808a0ae81c-kube-api-access-95p8b\") pod \"test-curl-7407-140271940103312\" (UID: \"6a45fe8b-da9f-4447-a749-f1808a0ae81c\") " pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:48.527394 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.527372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95p8b\" (UniqueName: \"kubernetes.io/projected/6a45fe8b-da9f-4447-a749-f1808a0ae81c-kube-api-access-95p8b\") pod \"test-curl-7407-140271940103312\" (UID: \"6a45fe8b-da9f-4447-a749-f1808a0ae81c\") " pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:48.659651 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.659596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:48.787616 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:48.787590 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:48.789786 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:36:48.789746 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a45fe8b_da9f_4447_a749_f1808a0ae81c.slice/crio-6c4198ac69b51de5709dc0bb0d8d68c501ecf0ef634c2c9e4d8f480a161e3f3b WatchSource:0}: Error finding container 6c4198ac69b51de5709dc0bb0d8d68c501ecf0ef634c2c9e4d8f480a161e3f3b: Status 404 returned error can't find the container with id 6c4198ac69b51de5709dc0bb0d8d68c501ecf0ef634c2c9e4d8f480a161e3f3b Jul 24 16:36:49.047211 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:49.047186 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:49.047372 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:49.047193 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5390f967d2659b87715bb7a4d72fe2c9d4254b5a29b6829c39095a92f17eab8e" Jul 24 16:36:49.048476 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:49.048452 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271940103312" event={"ID":"6a45fe8b-da9f-4447-a749-f1808a0ae81c","Type":"ContainerStarted","Data":"6c4198ac69b51de5709dc0bb0d8d68c501ecf0ef634c2c9e4d8f480a161e3f3b"} Jul 24 16:36:49.058388 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:49.058362 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7407-140271940103312" oldPodUID="c64e1148-79c9-4b6c-9cfb-d139e10e4222" podUID="6a45fe8b-da9f-4447-a749-f1808a0ae81c" Jul 24 16:36:50.010805 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:50.010770 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c64e1148-79c9-4b6c-9cfb-d139e10e4222" path="/var/lib/kubelet/pods/c64e1148-79c9-4b6c-9cfb-d139e10e4222/volumes" Jul 24 16:36:50.054758 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:50.054731 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a45fe8b-da9f-4447-a749-f1808a0ae81c" containerID="69762a684e097b2f8bb407001247792643354589fbeb63fbb42250c1eb2d02c8" exitCode=0 Jul 24 16:36:50.054908 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:50.054777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271940103312" event={"ID":"6a45fe8b-da9f-4447-a749-f1808a0ae81c","Type":"ContainerDied","Data":"69762a684e097b2f8bb407001247792643354589fbeb63fbb42250c1eb2d02c8"} Jul 24 16:36:51.197152 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.197125 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:51.214986 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.214963 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7407-140271940103312_6a45fe8b-da9f-4447-a749-f1808a0ae81c/test-curl-7407-140271940103312/0.log" Jul 24 16:36:51.239971 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.239928 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:51.242216 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.242193 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7407-140271940103312"] Jul 24 16:36:51.344001 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.343904 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95p8b\" (UniqueName: \"kubernetes.io/projected/6a45fe8b-da9f-4447-a749-f1808a0ae81c-kube-api-access-95p8b\") pod \"6a45fe8b-da9f-4447-a749-f1808a0ae81c\" (UID: \"6a45fe8b-da9f-4447-a749-f1808a0ae81c\") " Jul 24 16:36:51.346056 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.346036 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a45fe8b-da9f-4447-a749-f1808a0ae81c-kube-api-access-95p8b" (OuterVolumeSpecName: "kube-api-access-95p8b") pod "6a45fe8b-da9f-4447-a749-f1808a0ae81c" (UID: "6a45fe8b-da9f-4447-a749-f1808a0ae81c"). InnerVolumeSpecName "kube-api-access-95p8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:36:51.375641 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.375616 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:51.376031 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.376018 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a45fe8b-da9f-4447-a749-f1808a0ae81c" containerName="test-curl-7407-140271940103312" Jul 24 16:36:51.376093 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.376032 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a45fe8b-da9f-4447-a749-f1808a0ae81c" containerName="test-curl-7407-140271940103312" Jul 24 16:36:51.376128 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.376094 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a45fe8b-da9f-4447-a749-f1808a0ae81c" containerName="test-curl-7407-140271940103312" Jul 24 16:36:51.379110 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.379097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:51.386573 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.386550 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:51.445352 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.445319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbk6\" (UniqueName: \"kubernetes.io/projected/ff895e91-ad2d-4a4b-89b4-c3c354d87f36-kube-api-access-vtbk6\") pod \"test-curl-7407-140271939680128\" (UID: \"ff895e91-ad2d-4a4b-89b4-c3c354d87f36\") " pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:51.445515 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.445470 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95p8b\" (UniqueName: \"kubernetes.io/projected/6a45fe8b-da9f-4447-a749-f1808a0ae81c-kube-api-access-95p8b\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:36:51.546010 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.545982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbk6\" (UniqueName: \"kubernetes.io/projected/ff895e91-ad2d-4a4b-89b4-c3c354d87f36-kube-api-access-vtbk6\") pod \"test-curl-7407-140271939680128\" (UID: \"ff895e91-ad2d-4a4b-89b4-c3c354d87f36\") " pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:51.553610 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.553584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbk6\" (UniqueName: \"kubernetes.io/projected/ff895e91-ad2d-4a4b-89b4-c3c354d87f36-kube-api-access-vtbk6\") pod \"test-curl-7407-140271939680128\" (UID: \"ff895e91-ad2d-4a4b-89b4-c3c354d87f36\") " pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:51.689008 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.688901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:51.822256 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:51.822224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:51.825496 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:36:51.825473 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff895e91_ad2d_4a4b_89b4_c3c354d87f36.slice/crio-30ba89839f191c425569973cd4412a5b8ab6c5a76999847c67e793834eb1c693 WatchSource:0}: Error finding container 30ba89839f191c425569973cd4412a5b8ab6c5a76999847c67e793834eb1c693: Status 404 returned error can't find the container with id 30ba89839f191c425569973cd4412a5b8ab6c5a76999847c67e793834eb1c693 Jul 24 16:36:52.017267 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:52.017232 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a45fe8b-da9f-4447-a749-f1808a0ae81c" path="/var/lib/kubelet/pods/6a45fe8b-da9f-4447-a749-f1808a0ae81c/volumes" Jul 24 16:36:52.066746 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:52.066724 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271940103312" Jul 24 16:36:52.066911 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:52.066748 2580 scope.go:117] "RemoveContainer" containerID="69762a684e097b2f8bb407001247792643354589fbeb63fbb42250c1eb2d02c8" Jul 24 16:36:52.068178 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:52.068152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939680128" event={"ID":"ff895e91-ad2d-4a4b-89b4-c3c354d87f36","Type":"ContainerStarted","Data":"30ba89839f191c425569973cd4412a5b8ab6c5a76999847c67e793834eb1c693"} Jul 24 16:36:53.075996 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:53.075962 2580 generic.go:356] "Generic (PLEG): container finished" podID="ff895e91-ad2d-4a4b-89b4-c3c354d87f36" containerID="059cfd5a2a9b76be6fcba0acb914e7dc3d92c5f37c64c3c6bce7343a554afccc" exitCode=0 Jul 24 16:36:53.076361 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:53.076005 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939680128" event={"ID":"ff895e91-ad2d-4a4b-89b4-c3c354d87f36","Type":"ContainerDied","Data":"059cfd5a2a9b76be6fcba0acb914e7dc3d92c5f37c64c3c6bce7343a554afccc"} Jul 24 16:36:54.209600 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.209579 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:54.227209 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.227178 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7407-140271939680128_ff895e91-ad2d-4a4b-89b4-c3c354d87f36/test-curl-7407-140271939680128/0.log" Jul 24 16:36:54.250601 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.250578 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:54.252321 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.252301 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:54.272435 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.272413 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtbk6\" (UniqueName: \"kubernetes.io/projected/ff895e91-ad2d-4a4b-89b4-c3c354d87f36-kube-api-access-vtbk6\") pod \"ff895e91-ad2d-4a4b-89b4-c3c354d87f36\" (UID: \"ff895e91-ad2d-4a4b-89b4-c3c354d87f36\") " Jul 24 16:36:54.274518 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.274494 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff895e91-ad2d-4a4b-89b4-c3c354d87f36-kube-api-access-vtbk6" (OuterVolumeSpecName: "kube-api-access-vtbk6") pod "ff895e91-ad2d-4a4b-89b4-c3c354d87f36" (UID: "ff895e91-ad2d-4a4b-89b4-c3c354d87f36"). InnerVolumeSpecName "kube-api-access-vtbk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:36:54.369815 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.369757 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:54.370155 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.370142 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff895e91-ad2d-4a4b-89b4-c3c354d87f36" containerName="test-curl-7407-140271939680128" Jul 24 16:36:54.370196 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.370157 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff895e91-ad2d-4a4b-89b4-c3c354d87f36" containerName="test-curl-7407-140271939680128" Jul 24 16:36:54.370240 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.370231 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff895e91-ad2d-4a4b-89b4-c3c354d87f36" containerName="test-curl-7407-140271939680128" Jul 24 16:36:54.373194 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.373180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:54.373334 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.373317 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtbk6\" (UniqueName: \"kubernetes.io/projected/ff895e91-ad2d-4a4b-89b4-c3c354d87f36-kube-api-access-vtbk6\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:36:54.379325 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.379303 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:54.474060 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.474030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v87q\" (UniqueName: \"kubernetes.io/projected/a0eb25d6-638b-42f3-9e88-f40be2a95a7f-kube-api-access-5v87q\") pod \"test-curl-7407-140271939680128\" (UID: \"a0eb25d6-638b-42f3-9e88-f40be2a95a7f\") " pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:54.574705 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.574671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v87q\" (UniqueName: \"kubernetes.io/projected/a0eb25d6-638b-42f3-9e88-f40be2a95a7f-kube-api-access-5v87q\") pod \"test-curl-7407-140271939680128\" (UID: \"a0eb25d6-638b-42f3-9e88-f40be2a95a7f\") " pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:54.582761 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.582732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v87q\" (UniqueName: \"kubernetes.io/projected/a0eb25d6-638b-42f3-9e88-f40be2a95a7f-kube-api-access-5v87q\") pod \"test-curl-7407-140271939680128\" (UID: \"a0eb25d6-638b-42f3-9e88-f40be2a95a7f\") " pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:54.683097 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.683037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:54.813562 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:54.813537 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:54.816197 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:36:54.816163 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0eb25d6_638b_42f3_9e88_f40be2a95a7f.slice/crio-95127098e3a166ee6b4ba113977c9474ceb47773086a496d566b8426e9053f84 WatchSource:0}: Error finding container 95127098e3a166ee6b4ba113977c9474ceb47773086a496d566b8426e9053f84: Status 404 returned error can't find the container with id 95127098e3a166ee6b4ba113977c9474ceb47773086a496d566b8426e9053f84 Jul 24 16:36:55.086135 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:55.086108 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:55.086307 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:55.086117 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30ba89839f191c425569973cd4412a5b8ab6c5a76999847c67e793834eb1c693" Jul 24 16:36:55.087280 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:55.087256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939680128" event={"ID":"a0eb25d6-638b-42f3-9e88-f40be2a95a7f","Type":"ContainerStarted","Data":"95127098e3a166ee6b4ba113977c9474ceb47773086a496d566b8426e9053f84"} Jul 24 16:36:55.098316 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:55.098291 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7407-140271939680128" oldPodUID="ff895e91-ad2d-4a4b-89b4-c3c354d87f36" podUID="a0eb25d6-638b-42f3-9e88-f40be2a95a7f" Jul 24 16:36:56.010765 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:56.010733 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff895e91-ad2d-4a4b-89b4-c3c354d87f36" path="/var/lib/kubelet/pods/ff895e91-ad2d-4a4b-89b4-c3c354d87f36/volumes" Jul 24 16:36:56.093410 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:56.093374 2580 generic.go:356] "Generic (PLEG): container finished" podID="a0eb25d6-638b-42f3-9e88-f40be2a95a7f" containerID="f7326bbde074d5ccc0946dcf483adc50ae8739369e95a624da29675303a9e677" exitCode=0 Jul 24 16:36:56.093577 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:56.093466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7407-140271939680128" event={"ID":"a0eb25d6-638b-42f3-9e88-f40be2a95a7f","Type":"ContainerDied","Data":"f7326bbde074d5ccc0946dcf483adc50ae8739369e95a624da29675303a9e677"} Jul 24 16:36:57.234770 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:57.234746 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:57.253991 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:57.253923 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7407-140271939680128_a0eb25d6-638b-42f3-9e88-f40be2a95a7f/test-curl-7407-140271939680128/0.log" Jul 24 16:36:57.286388 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:57.286352 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:57.288138 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:57.288116 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7407-140271939680128"] Jul 24 16:36:57.300654 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:57.300631 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v87q\" (UniqueName: \"kubernetes.io/projected/a0eb25d6-638b-42f3-9e88-f40be2a95a7f-kube-api-access-5v87q\") pod \"a0eb25d6-638b-42f3-9e88-f40be2a95a7f\" (UID: \"a0eb25d6-638b-42f3-9e88-f40be2a95a7f\") " Jul 24 16:36:57.302797 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:57.302772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0eb25d6-638b-42f3-9e88-f40be2a95a7f-kube-api-access-5v87q" (OuterVolumeSpecName: "kube-api-access-5v87q") pod "a0eb25d6-638b-42f3-9e88-f40be2a95a7f" (UID: "a0eb25d6-638b-42f3-9e88-f40be2a95a7f"). InnerVolumeSpecName "kube-api-access-5v87q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:36:57.401553 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:57.401473 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v87q\" (UniqueName: \"kubernetes.io/projected/a0eb25d6-638b-42f3-9e88-f40be2a95a7f-kube-api-access-5v87q\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:36:58.013417 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:58.013382 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0eb25d6-638b-42f3-9e88-f40be2a95a7f" path="/var/lib/kubelet/pods/a0eb25d6-638b-42f3-9e88-f40be2a95a7f/volumes" Jul 24 16:36:58.104958 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:58.104902 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7407-140271939680128" Jul 24 16:36:58.105147 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:36:58.104907 2580 scope.go:117] "RemoveContainer" containerID="f7326bbde074d5ccc0946dcf483adc50ae8739369e95a624da29675303a9e677" Jul 24 16:37:01.076952 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.076906 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw"] Jul 24 16:37:01.077545 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.077527 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0eb25d6-638b-42f3-9e88-f40be2a95a7f" containerName="test-curl-7407-140271939680128" Jul 24 16:37:01.077603 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.077549 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0eb25d6-638b-42f3-9e88-f40be2a95a7f" containerName="test-curl-7407-140271939680128" Jul 24 16:37:01.077668 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.077655 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0eb25d6-638b-42f3-9e88-f40be2a95a7f" containerName="test-curl-7407-140271939680128" Jul 24 16:37:01.082527 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.082501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.085501 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.085466 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ae4522-serving-cert\"" Jul 24 16:37:01.104566 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.104536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw"] Jul 24 16:37:01.233011 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.232979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c027647-2972-49fe-a351-8e2981d825d6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.233185 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.233032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2lj\" (UniqueName: \"kubernetes.io/projected/1c027647-2972-49fe-a351-8e2981d825d6-kube-api-access-wh2lj\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.233185 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.233090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c027647-2972-49fe-a351-8e2981d825d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.333865 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.333786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2lj\" (UniqueName: \"kubernetes.io/projected/1c027647-2972-49fe-a351-8e2981d825d6-kube-api-access-wh2lj\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.333865 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.333831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c027647-2972-49fe-a351-8e2981d825d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.334125 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.333905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c027647-2972-49fe-a351-8e2981d825d6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.334571 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.334543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c027647-2972-49fe-a351-8e2981d825d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.336442 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.336425 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c027647-2972-49fe-a351-8e2981d825d6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.341656 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.341631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2lj\" (UniqueName: \"kubernetes.io/projected/1c027647-2972-49fe-a351-8e2981d825d6-kube-api-access-wh2lj\") pod \"maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.393727 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.393700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:01.533971 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:01.533922 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw"] Jul 24 16:37:01.535907 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:37:01.535874 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c027647_2972_49fe_a351_8e2981d825d6.slice/crio-9a54c30d327b56c27998ba3bed22db02c35e2bf7b2a6dbd92424584bca7bcfb9 WatchSource:0}: Error finding container 9a54c30d327b56c27998ba3bed22db02c35e2bf7b2a6dbd92424584bca7bcfb9: Status 404 returned error can't find the container with id 9a54c30d327b56c27998ba3bed22db02c35e2bf7b2a6dbd92424584bca7bcfb9 Jul 24 16:37:02.127963 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:02.127917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" event={"ID":"1c027647-2972-49fe-a351-8e2981d825d6","Type":"ContainerStarted","Data":"9a54c30d327b56c27998ba3bed22db02c35e2bf7b2a6dbd92424584bca7bcfb9"} Jul 24 16:37:03.135132 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:03.135084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" event={"ID":"1c027647-2972-49fe-a351-8e2981d825d6","Type":"ContainerStarted","Data":"ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea"} Jul 24 16:37:03.135556 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:03.135193 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:03.154994 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:03.154952 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" podStartSLOduration=1.624251579 podStartE2EDuration="2.154923062s" totalImagesPullingTime="530.671483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:37:03.152289566 +0000 UTC m=+2991.673898106" watchObservedRunningTime="2026-07-24 16:37:03.154923062 +0000 UTC m=+2991.676531602" Jul 24 16:37:09.147154 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:09.147130 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:37:12.188107 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:12.188081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:37:12.197017 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:12.196992 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:37:20.691179 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.691140 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k"] Jul 24 16:37:20.697378 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.697356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.699496 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.699464 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d71769-serving-cert\"" Jul 24 16:37:20.705633 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.705590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnnv\" (UniqueName: \"kubernetes.io/projected/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-kube-api-access-wtnnv\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.705749 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.705659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.705749 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.705726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.706684 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.706622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k"] Jul 24 16:37:20.806273 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.806225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.806474 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.806313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnnv\" (UniqueName: \"kubernetes.io/projected/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-kube-api-access-wtnnv\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.806474 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.806354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.806474 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:37:20.806463 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-serving-cert: secret "maas-api-e2e-ipp-b-d71769-serving-cert" not found Jul 24 16:37:20.806651 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:37:20.806536 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls podName:4d31e263-38c6-4cb9-881d-4f8a1e67ddd7 nodeName:}" failed. No retries permitted until 2026-07-24 16:37:21.306520725 +0000 UTC m=+3009.828129243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls") pod "maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" (UID: "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7") : secret "maas-api-e2e-ipp-b-d71769-serving-cert" not found Jul 24 16:37:20.806928 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.806905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:20.814731 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:20.814706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnnv\" (UniqueName: \"kubernetes.io/projected/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-kube-api-access-wtnnv\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:21.310915 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:21.310878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:21.311106 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:37:21.311031 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-serving-cert: secret "maas-api-e2e-ipp-b-d71769-serving-cert" not found Jul 24 16:37:21.311153 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:37:21.311118 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls podName:4d31e263-38c6-4cb9-881d-4f8a1e67ddd7 nodeName:}" failed. No retries permitted until 2026-07-24 16:37:22.311099344 +0000 UTC m=+3010.832707862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls") pod "maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" (UID: "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7") : secret "maas-api-e2e-ipp-b-d71769-serving-cert" not found Jul 24 16:37:22.320208 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:22.320172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:22.322831 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:22.322807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:22.509555 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:22.509522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:22.653255 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:22.653226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k"] Jul 24 16:37:22.655314 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:37:22.655287 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d31e263_38c6_4cb9_881d_4f8a1e67ddd7.slice/crio-59619013bb79e80b27b39a7395cc08b3c7dc6291cf072efd4e53c200d8ee4cb5 WatchSource:0}: Error finding container 59619013bb79e80b27b39a7395cc08b3c7dc6291cf072efd4e53c200d8ee4cb5: Status 404 returned error can't find the container with id 59619013bb79e80b27b39a7395cc08b3c7dc6291cf072efd4e53c200d8ee4cb5 Jul 24 16:37:23.237404 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:23.237376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" event={"ID":"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7","Type":"ContainerStarted","Data":"59619013bb79e80b27b39a7395cc08b3c7dc6291cf072efd4e53c200d8ee4cb5"} Jul 24 16:37:24.244403 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:24.244368 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" event={"ID":"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7","Type":"ContainerStarted","Data":"1506cb76fcc8a1190bf89548780fe3747290d6fab60aade798920c109d8c2a08"} Jul 24 16:37:24.244796 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:24.244448 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:24.263162 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:24.263117 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" podStartSLOduration=3.710586628 podStartE2EDuration="4.263100882s" totalImagesPullingTime="552.514254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:37:24.261606673 +0000 UTC m=+3012.783215225" watchObservedRunningTime="2026-07-24 16:37:24.263100882 +0000 UTC m=+3012.784709428" Jul 24 16:37:30.255872 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:30.255841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:37:41.704480 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:37:41.704451 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9czsr_aba87454-6133-46b3-a7f1-1f2053b04e98/payload-processing/0.log" Jul 24 16:38:28.823369 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:28.823290 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9czsr_aba87454-6133-46b3-a7f1-1f2053b04e98/payload-processing/0.log" Jul 24 16:38:34.903889 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:34.903856 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45"] Jul 24 16:38:34.908129 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:34.908106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:34.910251 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:34.910232 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-292d47-serving-cert\"" Jul 24 16:38:34.916460 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:34.916437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45"] Jul 24 16:38:34.979595 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:34.979563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb37dbf-f456-4882-87a1-11ea687417e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:34.979595 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:34.979590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxwj\" (UniqueName: \"kubernetes.io/projected/eeb37dbf-f456-4882-87a1-11ea687417e1-kube-api-access-njxwj\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:34.979785 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:34.979757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.080474 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.080448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb37dbf-f456-4882-87a1-11ea687417e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.080474 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.080476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxwj\" (UniqueName: \"kubernetes.io/projected/eeb37dbf-f456-4882-87a1-11ea687417e1-kube-api-access-njxwj\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.080687 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.080669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.080777 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:38:35.080765 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-serving-cert: secret "maas-api-e2e-ipp-cleanup-292d47-serving-cert" not found Jul 24 16:38:35.080846 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:38:35.080819 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls podName:eeb37dbf-f456-4882-87a1-11ea687417e1 nodeName:}" failed. No retries permitted until 2026-07-24 16:38:35.580801144 +0000 UTC m=+3084.102409679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" (UID: "eeb37dbf-f456-4882-87a1-11ea687417e1") : secret "maas-api-e2e-ipp-cleanup-292d47-serving-cert" not found Jul 24 16:38:35.081235 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.081214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb37dbf-f456-4882-87a1-11ea687417e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.090417 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.090396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxwj\" (UniqueName: \"kubernetes.io/projected/eeb37dbf-f456-4882-87a1-11ea687417e1-kube-api-access-njxwj\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.586185 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.586149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.589084 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.589060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.820028 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.819994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:35.955399 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:35.955327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45"] Jul 24 16:38:35.957810 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:38:35.957785 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb37dbf_f456_4882_87a1_11ea687417e1.slice/crio-92cec4c07d874920d253d40c66ada46899edec022ee1b130b28af3511249d0c2 WatchSource:0}: Error finding container 92cec4c07d874920d253d40c66ada46899edec022ee1b130b28af3511249d0c2: Status 404 returned error can't find the container with id 92cec4c07d874920d253d40c66ada46899edec022ee1b130b28af3511249d0c2 Jul 24 16:38:36.615294 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:36.615199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" event={"ID":"eeb37dbf-f456-4882-87a1-11ea687417e1","Type":"ContainerStarted","Data":"a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a"} Jul 24 16:38:36.615294 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:36.615253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" event={"ID":"eeb37dbf-f456-4882-87a1-11ea687417e1","Type":"ContainerStarted","Data":"92cec4c07d874920d253d40c66ada46899edec022ee1b130b28af3511249d0c2"} Jul 24 16:38:36.615510 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:36.615414 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:38:36.633424 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:36.633378 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" podStartSLOduration=2.2713072309999998 podStartE2EDuration="2.633362825s" totalImagesPullingTime="362.055594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:38:36.631757859 +0000 UTC m=+3085.153366400" watchObservedRunningTime="2026-07-24 16:38:36.633362825 +0000 UTC m=+3085.154971365" Jul 24 16:38:42.626879 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:38:42.626850 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:39:04.832164 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:04.832133 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45"] Jul 24 16:39:04.832655 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:04.832372 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" podUID="eeb37dbf-f456-4882-87a1-11ea687417e1" containerName="maas-api" containerID="cri-o://a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a" gracePeriod=30 Jul 24 16:39:05.092066 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.092006 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:39:05.158485 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.158451 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njxwj\" (UniqueName: \"kubernetes.io/projected/eeb37dbf-f456-4882-87a1-11ea687417e1-kube-api-access-njxwj\") pod \"eeb37dbf-f456-4882-87a1-11ea687417e1\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " Jul 24 16:39:05.158675 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.158565 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb37dbf-f456-4882-87a1-11ea687417e1-openshift-service-ca\") pod \"eeb37dbf-f456-4882-87a1-11ea687417e1\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " Jul 24 16:39:05.158675 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.158614 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls\") pod \"eeb37dbf-f456-4882-87a1-11ea687417e1\" (UID: \"eeb37dbf-f456-4882-87a1-11ea687417e1\") " Jul 24 16:39:05.159058 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.159025 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb37dbf-f456-4882-87a1-11ea687417e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eeb37dbf-f456-4882-87a1-11ea687417e1" (UID: "eeb37dbf-f456-4882-87a1-11ea687417e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:39:05.160887 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.160856 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eeb37dbf-f456-4882-87a1-11ea687417e1" (UID: "eeb37dbf-f456-4882-87a1-11ea687417e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:39:05.160887 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.160868 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb37dbf-f456-4882-87a1-11ea687417e1-kube-api-access-njxwj" (OuterVolumeSpecName: "kube-api-access-njxwj") pod "eeb37dbf-f456-4882-87a1-11ea687417e1" (UID: "eeb37dbf-f456-4882-87a1-11ea687417e1"). InnerVolumeSpecName "kube-api-access-njxwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:39:05.260038 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.260006 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxwj\" (UniqueName: \"kubernetes.io/projected/eeb37dbf-f456-4882-87a1-11ea687417e1-kube-api-access-njxwj\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:39:05.260038 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.260031 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb37dbf-f456-4882-87a1-11ea687417e1-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:39:05.260038 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.260040 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb37dbf-f456-4882-87a1-11ea687417e1-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:39:05.772981 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.772920 2580 generic.go:356] "Generic (PLEG): container finished" podID="eeb37dbf-f456-4882-87a1-11ea687417e1" containerID="a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a" exitCode=0 Jul 24 16:39:05.772981 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.772979 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" Jul 24 16:39:05.773204 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.773013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" event={"ID":"eeb37dbf-f456-4882-87a1-11ea687417e1","Type":"ContainerDied","Data":"a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a"} Jul 24 16:39:05.773204 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.773067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45" event={"ID":"eeb37dbf-f456-4882-87a1-11ea687417e1","Type":"ContainerDied","Data":"92cec4c07d874920d253d40c66ada46899edec022ee1b130b28af3511249d0c2"} Jul 24 16:39:05.773204 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.773089 2580 scope.go:117] "RemoveContainer" containerID="a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a" Jul 24 16:39:05.782705 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.782691 2580 scope.go:117] "RemoveContainer" containerID="a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a" Jul 24 16:39:05.782979 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:39:05.782962 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a\": container with ID starting with a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a not found: ID does not exist" containerID="a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a" Jul 24 16:39:05.783037 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.782987 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a"} err="failed to get container status \"a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a\": rpc error: code = NotFound desc = could not find container \"a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a\": container with ID starting with a83c1fa6fd59aa1542e4cc97a0b1d0a407b4e5874a859bcc2436e979ed355c6a not found: ID does not exist" Jul 24 16:39:05.793216 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.793190 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45"] Jul 24 16:39:05.796886 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:05.796864 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-292d47-7d4d8b9854-x4p45"] Jul 24 16:39:06.010438 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:06.010405 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb37dbf-f456-4882-87a1-11ea687417e1" path="/var/lib/kubelet/pods/eeb37dbf-f456-4882-87a1-11ea687417e1/volumes" Jul 24 16:39:18.188911 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.188879 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klvvv"] Jul 24 16:39:18.189321 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.189306 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb37dbf-f456-4882-87a1-11ea687417e1" containerName="maas-api" Jul 24 16:39:18.189366 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.189323 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb37dbf-f456-4882-87a1-11ea687417e1" containerName="maas-api" Jul 24 16:39:18.189412 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.189403 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb37dbf-f456-4882-87a1-11ea687417e1" containerName="maas-api" Jul 24 16:39:18.194000 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.193986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:18.195873 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.195853 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-hhdnq\"" Jul 24 16:39:18.198979 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.198955 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klvvv"] Jul 24 16:39:18.284665 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.284634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbd6\" (UniqueName: \"kubernetes.io/projected/777497da-3902-4c09-adc4-a6f65ebfd976-kube-api-access-rdbd6\") pod \"kuadrant-operator-catalog-klvvv\" (UID: \"777497da-3902-4c09-adc4-a6f65ebfd976\") " pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:18.385532 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.385505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbd6\" (UniqueName: \"kubernetes.io/projected/777497da-3902-4c09-adc4-a6f65ebfd976-kube-api-access-rdbd6\") pod \"kuadrant-operator-catalog-klvvv\" (UID: \"777497da-3902-4c09-adc4-a6f65ebfd976\") " pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:18.396512 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.396480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbd6\" (UniqueName: \"kubernetes.io/projected/777497da-3902-4c09-adc4-a6f65ebfd976-kube-api-access-rdbd6\") pod \"kuadrant-operator-catalog-klvvv\" (UID: \"777497da-3902-4c09-adc4-a6f65ebfd976\") " pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:18.503773 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.503700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:18.639517 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.639494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klvvv"] Jul 24 16:39:18.642085 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:39:18.642057 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777497da_3902_4c09_adc4_a6f65ebfd976.slice/crio-3a14bd24c3a3717dc3088fa00f6fbbef4b1305f27bc716d01559593d0396bcb2 WatchSource:0}: Error finding container 3a14bd24c3a3717dc3088fa00f6fbbef4b1305f27bc716d01559593d0396bcb2: Status 404 returned error can't find the container with id 3a14bd24c3a3717dc3088fa00f6fbbef4b1305f27bc716d01559593d0396bcb2 Jul 24 16:39:18.846556 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:18.846515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" event={"ID":"777497da-3902-4c09-adc4-a6f65ebfd976","Type":"ContainerStarted","Data":"3a14bd24c3a3717dc3088fa00f6fbbef4b1305f27bc716d01559593d0396bcb2"} Jul 24 16:39:21.866662 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:21.866623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" event={"ID":"777497da-3902-4c09-adc4-a6f65ebfd976","Type":"ContainerStarted","Data":"69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274"} Jul 24 16:39:21.881992 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:21.881931 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" podStartSLOduration=1.470880432 podStartE2EDuration="3.881915845s" totalImagesPullingTime="2.411035413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:39:21.880233806 +0000 UTC m=+3130.401842379" watchObservedRunningTime="2026-07-24 16:39:21.881915845 +0000 UTC m=+3130.403524385" Jul 24 16:39:28.504353 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:28.504321 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:28.504807 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:28.504380 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:28.526478 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:28.526450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:28.923773 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:28.923744 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:28.954270 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:28.954235 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klvvv"] Jul 24 16:39:30.911507 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:30.911446 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" podUID="777497da-3902-4c09-adc4-a6f65ebfd976" containerName="registry-server" containerID="cri-o://69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274" gracePeriod=2 Jul 24 16:39:31.177237 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.177182 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:31.205710 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.205686 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdbd6\" (UniqueName: \"kubernetes.io/projected/777497da-3902-4c09-adc4-a6f65ebfd976-kube-api-access-rdbd6\") pod \"777497da-3902-4c09-adc4-a6f65ebfd976\" (UID: \"777497da-3902-4c09-adc4-a6f65ebfd976\") " Jul 24 16:39:31.207813 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.207793 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777497da-3902-4c09-adc4-a6f65ebfd976-kube-api-access-rdbd6" (OuterVolumeSpecName: "kube-api-access-rdbd6") pod "777497da-3902-4c09-adc4-a6f65ebfd976" (UID: "777497da-3902-4c09-adc4-a6f65ebfd976"). InnerVolumeSpecName "kube-api-access-rdbd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:39:31.306546 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.306518 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbd6\" (UniqueName: \"kubernetes.io/projected/777497da-3902-4c09-adc4-a6f65ebfd976-kube-api-access-rdbd6\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:39:31.917103 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.917070 2580 generic.go:356] "Generic (PLEG): container finished" podID="777497da-3902-4c09-adc4-a6f65ebfd976" containerID="69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274" exitCode=0 Jul 24 16:39:31.917572 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.917119 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" Jul 24 16:39:31.917572 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.917158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" event={"ID":"777497da-3902-4c09-adc4-a6f65ebfd976","Type":"ContainerDied","Data":"69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274"} Jul 24 16:39:31.917572 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.917214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-klvvv" event={"ID":"777497da-3902-4c09-adc4-a6f65ebfd976","Type":"ContainerDied","Data":"3a14bd24c3a3717dc3088fa00f6fbbef4b1305f27bc716d01559593d0396bcb2"} Jul 24 16:39:31.917572 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.917248 2580 scope.go:117] "RemoveContainer" containerID="69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274" Jul 24 16:39:31.935836 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.935817 2580 scope.go:117] "RemoveContainer" containerID="69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274" Jul 24 16:39:31.936164 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:39:31.936138 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274\": container with ID starting with 69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274 not found: ID does not exist" containerID="69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274" Jul 24 16:39:31.936232 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.936173 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274"} err="failed to get container status \"69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274\": rpc error: code = NotFound desc = could not find container \"69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274\": container with ID starting with 69e98d9652db57bb425ba0f7db8536af158aaeb0b78598d2c2c4cf6b662ba274 not found: ID does not exist" Jul 24 16:39:31.941278 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.941257 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klvvv"] Jul 24 16:39:31.946457 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:31.946434 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-klvvv"] Jul 24 16:39:32.010891 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:32.010865 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777497da-3902-4c09-adc4-a6f65ebfd976" path="/var/lib/kubelet/pods/777497da-3902-4c09-adc4-a6f65ebfd976/volumes" Jul 24 16:39:36.831513 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:36.831431 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw"] Jul 24 16:39:36.832054 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:36.831698 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" podUID="1c027647-2972-49fe-a351-8e2981d825d6" containerName="maas-api" containerID="cri-o://ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea" gracePeriod=30 Jul 24 16:39:37.086148 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.086095 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:39:37.160985 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.160958 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh2lj\" (UniqueName: \"kubernetes.io/projected/1c027647-2972-49fe-a351-8e2981d825d6-kube-api-access-wh2lj\") pod \"1c027647-2972-49fe-a351-8e2981d825d6\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " Jul 24 16:39:37.161138 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.161007 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c027647-2972-49fe-a351-8e2981d825d6-openshift-service-ca\") pod \"1c027647-2972-49fe-a351-8e2981d825d6\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " Jul 24 16:39:37.161138 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.161053 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c027647-2972-49fe-a351-8e2981d825d6-maas-api-tls\") pod \"1c027647-2972-49fe-a351-8e2981d825d6\" (UID: \"1c027647-2972-49fe-a351-8e2981d825d6\") " Jul 24 16:39:37.161466 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.161433 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c027647-2972-49fe-a351-8e2981d825d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c027647-2972-49fe-a351-8e2981d825d6" (UID: "1c027647-2972-49fe-a351-8e2981d825d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:39:37.163275 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.163249 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c027647-2972-49fe-a351-8e2981d825d6-kube-api-access-wh2lj" (OuterVolumeSpecName: "kube-api-access-wh2lj") pod "1c027647-2972-49fe-a351-8e2981d825d6" (UID: "1c027647-2972-49fe-a351-8e2981d825d6"). InnerVolumeSpecName "kube-api-access-wh2lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:39:37.163380 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.163323 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c027647-2972-49fe-a351-8e2981d825d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c027647-2972-49fe-a351-8e2981d825d6" (UID: "1c027647-2972-49fe-a351-8e2981d825d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:39:37.261821 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.261795 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2lj\" (UniqueName: \"kubernetes.io/projected/1c027647-2972-49fe-a351-8e2981d825d6-kube-api-access-wh2lj\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:39:37.261821 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.261817 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c027647-2972-49fe-a351-8e2981d825d6-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:39:37.261821 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.261827 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c027647-2972-49fe-a351-8e2981d825d6-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:39:37.948246 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.948219 2580 generic.go:356] "Generic (PLEG): container finished" podID="1c027647-2972-49fe-a351-8e2981d825d6" containerID="ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea" exitCode=0 Jul 24 16:39:37.948653 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.948268 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" Jul 24 16:39:37.948653 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.948292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" event={"ID":"1c027647-2972-49fe-a351-8e2981d825d6","Type":"ContainerDied","Data":"ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea"} Jul 24 16:39:37.948653 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.948332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw" event={"ID":"1c027647-2972-49fe-a351-8e2981d825d6","Type":"ContainerDied","Data":"9a54c30d327b56c27998ba3bed22db02c35e2bf7b2a6dbd92424584bca7bcfb9"} Jul 24 16:39:37.948653 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.948354 2580 scope.go:117] "RemoveContainer" containerID="ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea" Jul 24 16:39:37.962544 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.962516 2580 scope.go:117] "RemoveContainer" containerID="ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea" Jul 24 16:39:37.962834 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:39:37.962810 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea\": container with ID starting with ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea not found: ID does not exist" containerID="ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea" Jul 24 16:39:37.962916 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.962840 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea"} err="failed to get container status \"ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea\": rpc error: code = NotFound desc = could not find container \"ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea\": container with ID starting with ac93067fdff45955eeb6690e991f08b602522fb4898aab5882322c5286a5e4ea not found: ID does not exist" Jul 24 16:39:37.974210 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.974186 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw"] Jul 24 16:39:37.977515 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:37.977494 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae4522-746489b9cb-7g8zw"] Jul 24 16:39:38.010810 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:39:38.010785 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c027647-2972-49fe-a351-8e2981d825d6" path="/var/lib/kubelet/pods/1c027647-2972-49fe-a351-8e2981d825d6/volumes" Jul 24 16:40:46.121515 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.121468 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k"] Jul 24 16:40:46.122128 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.121862 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" podUID="4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" containerName="maas-api" containerID="cri-o://1506cb76fcc8a1190bf89548780fe3747290d6fab60aade798920c109d8c2a08" gracePeriod=30 Jul 24 16:40:46.292168 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.292139 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" containerID="1506cb76fcc8a1190bf89548780fe3747290d6fab60aade798920c109d8c2a08" exitCode=0 Jul 24 16:40:46.292339 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.292224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" event={"ID":"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7","Type":"ContainerDied","Data":"1506cb76fcc8a1190bf89548780fe3747290d6fab60aade798920c109d8c2a08"} Jul 24 16:40:46.371855 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.371796 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:40:46.510552 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.510522 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls\") pod \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " Jul 24 16:40:46.510734 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.510567 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtnnv\" (UniqueName: \"kubernetes.io/projected/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-kube-api-access-wtnnv\") pod \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " Jul 24 16:40:46.510734 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.510607 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-openshift-service-ca\") pod \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\" (UID: \"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7\") " Jul 24 16:40:46.511105 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.511075 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" (UID: "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:40:46.512741 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.512718 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" (UID: "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:40:46.512857 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.512837 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-kube-api-access-wtnnv" (OuterVolumeSpecName: "kube-api-access-wtnnv") pod "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" (UID: "4d31e263-38c6-4cb9-881d-4f8a1e67ddd7"). InnerVolumeSpecName "kube-api-access-wtnnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:40:46.611857 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.611829 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:40:46.611857 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.611851 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtnnv\" (UniqueName: \"kubernetes.io/projected/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-kube-api-access-wtnnv\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:40:46.611857 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:46.611861 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:40:47.299011 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:47.298971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" event={"ID":"4d31e263-38c6-4cb9-881d-4f8a1e67ddd7","Type":"ContainerDied","Data":"59619013bb79e80b27b39a7395cc08b3c7dc6291cf072efd4e53c200d8ee4cb5"} Jul 24 16:40:47.299426 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:47.299040 2580 scope.go:117] "RemoveContainer" containerID="1506cb76fcc8a1190bf89548780fe3747290d6fab60aade798920c109d8c2a08" Jul 24 16:40:47.299426 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:47.298990 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k" Jul 24 16:40:47.321654 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:47.321621 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k"] Jul 24 16:40:47.323539 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:47.323517 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71769-94f444cbb-gf59k"] Jul 24 16:40:48.011573 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:40:48.011540 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" path="/var/lib/kubelet/pods/4d31e263-38c6-4cb9-881d-4f8a1e67ddd7/volumes" Jul 24 16:41:05.914334 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:05.914241 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm"] Jul 24 16:41:05.914790 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:05.914533 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" podUID="cb97dad9-0828-41da-a20b-633d509f61b7" containerName="maas-api" containerID="cri-o://6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976" gracePeriod=30 Jul 24 16:41:06.152188 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.152163 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:41:06.181994 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.181898 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb97dad9-0828-41da-a20b-633d509f61b7-openshift-service-ca\") pod \"cb97dad9-0828-41da-a20b-633d509f61b7\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " Jul 24 16:41:06.181994 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.181957 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp5s4\" (UniqueName: \"kubernetes.io/projected/cb97dad9-0828-41da-a20b-633d509f61b7-kube-api-access-jp5s4\") pod \"cb97dad9-0828-41da-a20b-633d509f61b7\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " Jul 24 16:41:06.181994 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.181989 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb97dad9-0828-41da-a20b-633d509f61b7-maas-api-tls\") pod \"cb97dad9-0828-41da-a20b-633d509f61b7\" (UID: \"cb97dad9-0828-41da-a20b-633d509f61b7\") " Jul 24 16:41:06.182249 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.182224 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb97dad9-0828-41da-a20b-633d509f61b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb97dad9-0828-41da-a20b-633d509f61b7" (UID: "cb97dad9-0828-41da-a20b-633d509f61b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:41:06.184193 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.184163 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb97dad9-0828-41da-a20b-633d509f61b7-kube-api-access-jp5s4" (OuterVolumeSpecName: "kube-api-access-jp5s4") pod "cb97dad9-0828-41da-a20b-633d509f61b7" (UID: "cb97dad9-0828-41da-a20b-633d509f61b7"). InnerVolumeSpecName "kube-api-access-jp5s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:41:06.184331 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.184259 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb97dad9-0828-41da-a20b-633d509f61b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb97dad9-0828-41da-a20b-633d509f61b7" (UID: "cb97dad9-0828-41da-a20b-633d509f61b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:41:06.282899 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.282873 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb97dad9-0828-41da-a20b-633d509f61b7-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:41:06.282899 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.282897 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5s4\" (UniqueName: \"kubernetes.io/projected/cb97dad9-0828-41da-a20b-633d509f61b7-kube-api-access-jp5s4\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:41:06.283078 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.282907 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb97dad9-0828-41da-a20b-633d509f61b7-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:41:06.399067 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.399020 2580 generic.go:356] "Generic (PLEG): container finished" podID="cb97dad9-0828-41da-a20b-633d509f61b7" containerID="6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976" exitCode=0 Jul 24 16:41:06.399244 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.399086 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" Jul 24 16:41:06.399244 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.399088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" event={"ID":"cb97dad9-0828-41da-a20b-633d509f61b7","Type":"ContainerDied","Data":"6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976"} Jul 24 16:41:06.399244 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.399166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm" event={"ID":"cb97dad9-0828-41da-a20b-633d509f61b7","Type":"ContainerDied","Data":"272b4cf6b87a78505049a6eff040fbbe2a290f666e791dcf89089993e007c74f"} Jul 24 16:41:06.399244 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.399203 2580 scope.go:117] "RemoveContainer" containerID="6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976" Jul 24 16:41:06.408681 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.408667 2580 scope.go:117] "RemoveContainer" containerID="6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976" Jul 24 16:41:06.408930 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:41:06.408912 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976\": container with ID starting with 6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976 not found: ID does not exist" containerID="6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976" Jul 24 16:41:06.409067 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.409047 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976"} err="failed to get container status \"6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976\": rpc error: code = NotFound desc = could not find container \"6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976\": container with ID starting with 6d1d5f179655eeaf8e8ac2783979b4992fd0212ea1226f53015e9c8a7a8b3976 not found: ID does not exist" Jul 24 16:41:06.421556 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.421534 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm"] Jul 24 16:41:06.423376 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:06.423348 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a5fa08-78bd88d6c4-n9bqm"] Jul 24 16:41:08.011139 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:08.011101 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb97dad9-0828-41da-a20b-633d509f61b7" path="/var/lib/kubelet/pods/cb97dad9-0828-41da-a20b-633d509f61b7/volumes" Jul 24 16:41:29.546407 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.546373 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96"] Jul 24 16:41:29.547341 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.546644 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" podUID="92bc2101-6c00-43b7-99e0-1ff6cfbbf250" containerName="maas-api" containerID="cri-o://350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7" gracePeriod=30 Jul 24 16:41:29.804537 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.804475 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:41:29.898172 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.898143 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-openshift-service-ca\") pod \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " Jul 24 16:41:29.898330 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.898185 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls\") pod \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " Jul 24 16:41:29.898330 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.898228 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl4wr\" (UniqueName: \"kubernetes.io/projected/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-kube-api-access-tl4wr\") pod \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\" (UID: \"92bc2101-6c00-43b7-99e0-1ff6cfbbf250\") " Jul 24 16:41:29.898525 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.898500 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92bc2101-6c00-43b7-99e0-1ff6cfbbf250" (UID: "92bc2101-6c00-43b7-99e0-1ff6cfbbf250"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:41:29.900359 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.900331 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92bc2101-6c00-43b7-99e0-1ff6cfbbf250" (UID: "92bc2101-6c00-43b7-99e0-1ff6cfbbf250"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:41:29.900481 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.900457 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-kube-api-access-tl4wr" (OuterVolumeSpecName: "kube-api-access-tl4wr") pod "92bc2101-6c00-43b7-99e0-1ff6cfbbf250" (UID: "92bc2101-6c00-43b7-99e0-1ff6cfbbf250"). InnerVolumeSpecName "kube-api-access-tl4wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:41:29.999248 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.999221 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl4wr\" (UniqueName: \"kubernetes.io/projected/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-kube-api-access-tl4wr\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:41:29.999248 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.999249 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-openshift-service-ca\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:41:29.999430 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:29.999262 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92bc2101-6c00-43b7-99e0-1ff6cfbbf250-maas-api-tls\") on node \"ip-10-0-140-155.ec2.internal\" DevicePath \"\"" Jul 24 16:41:30.517972 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.517916 2580 generic.go:356] "Generic (PLEG): container finished" podID="92bc2101-6c00-43b7-99e0-1ff6cfbbf250" containerID="350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7" exitCode=0 Jul 24 16:41:30.518141 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.517974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" event={"ID":"92bc2101-6c00-43b7-99e0-1ff6cfbbf250","Type":"ContainerDied","Data":"350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7"} Jul 24 16:41:30.518141 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.518027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" event={"ID":"92bc2101-6c00-43b7-99e0-1ff6cfbbf250","Type":"ContainerDied","Data":"2aa7d9b992fb2423948b0b8a613ff8ab3e452ce3b5e1233983b445acb745cc8a"} Jul 24 16:41:30.518141 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.518052 2580 scope.go:117] "RemoveContainer" containerID="350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7" Jul 24 16:41:30.518141 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.517989 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96" Jul 24 16:41:30.529471 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.529403 2580 scope.go:117] "RemoveContainer" containerID="350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7" Jul 24 16:41:30.530024 ip-10-0-140-155 kubenswrapper[2580]: E0724 16:41:30.529962 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7\": container with ID starting with 350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7 not found: ID does not exist" containerID="350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7" Jul 24 16:41:30.530024 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.529998 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7"} err="failed to get container status \"350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7\": rpc error: code = NotFound desc = could not find container \"350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7\": container with ID starting with 350f197b94193ff7812b9bc3024a5341c97b4e5a82412e2fdff3db9477ca20d7 not found: ID does not exist" Jul 24 16:41:30.532571 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.532543 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96"] Jul 24 16:41:30.534588 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:30.534571 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-675edf-57757648fc-64d96"] Jul 24 16:41:32.010573 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:32.010535 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92bc2101-6c00-43b7-99e0-1ff6cfbbf250" path="/var/lib/kubelet/pods/92bc2101-6c00-43b7-99e0-1ff6cfbbf250/volumes" Jul 24 16:41:46.186486 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:46.186450 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cc959d7c4-vk82w_c754a2c8-8694-41c0-8b4f-bedd0be27e9f/manager/0.log" Jul 24 16:41:46.774048 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:46.774018 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749946f7d7-swn4q_7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3/manager/0.log" Jul 24 16:41:46.780686 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:46.780664 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749946f7d7-swn4q_7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3/copy-manifests/0.log" Jul 24 16:41:47.109579 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:47.109503 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6df5894bdc-v2rgf_2dfd6a9a-9c84-42c0-87d8-0364266c7403/maas-api/0.log" Jul 24 16:41:47.547759 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:47.547728 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lzq88_3879ebc7-7777-49ca-beff-6224945fa871/postgres/0.log" Jul 24 16:41:49.107342 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:49.107315 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-b7296_95bd019a-375b-44e2-84d9-0a306f8672a7/kuadrant-console-plugin/0.log" Jul 24 16:41:50.264155 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:50.264131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545786b4d4-v222p_f3cd4577-c073-49d2-ad94-70f3ace80f8e/kube-auth-proxy/0.log" Jul 24 16:41:50.625904 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:50.625827 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9czsr_aba87454-6133-46b3-a7f1-1f2053b04e98/payload-processing/0.log" Jul 24 16:41:51.182305 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:51.182270 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gpmdb_2c1669bd-c09d-4a14-afda-9c0d835cc0ce/main/0.log" Jul 24 16:41:51.516696 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:51.516666 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-s79jx_4de7c0b6-a191-4b7f-ba50-4e1ffc3511e7/main/0.log" Jul 24 16:41:51.625020 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:51.624998 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bvwmj_a35d3e38-cf86-4e65-926d-4244c0884234/main/0.log" Jul 24 16:41:56.806960 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.806899 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6ww8s/must-gather-lndkr"] Jul 24 16:41:56.807802 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807772 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bc2101-6c00-43b7-99e0-1ff6cfbbf250" containerName="maas-api" Jul 24 16:41:56.807802 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807803 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bc2101-6c00-43b7-99e0-1ff6cfbbf250" containerName="maas-api" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807834 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb97dad9-0828-41da-a20b-633d509f61b7" containerName="maas-api" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807843 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb97dad9-0828-41da-a20b-633d509f61b7" containerName="maas-api" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807870 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c027647-2972-49fe-a351-8e2981d825d6" containerName="maas-api" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807880 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c027647-2972-49fe-a351-8e2981d825d6" containerName="maas-api" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807890 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" containerName="maas-api" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807899 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" containerName="maas-api" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807927 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777497da-3902-4c09-adc4-a6f65ebfd976" containerName="registry-server" Jul 24 16:41:56.808014 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.807990 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="777497da-3902-4c09-adc4-a6f65ebfd976" containerName="registry-server" Jul 24 16:41:56.808497 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.808217 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92bc2101-6c00-43b7-99e0-1ff6cfbbf250" containerName="maas-api" Jul 24 16:41:56.808497 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.808240 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d31e263-38c6-4cb9-881d-4f8a1e67ddd7" containerName="maas-api" Jul 24 16:41:56.808497 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.808252 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c027647-2972-49fe-a351-8e2981d825d6" containerName="maas-api" Jul 24 16:41:56.808497 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.808261 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb97dad9-0828-41da-a20b-633d509f61b7" containerName="maas-api" Jul 24 16:41:56.808497 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.808278 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="777497da-3902-4c09-adc4-a6f65ebfd976" containerName="registry-server" Jul 24 16:41:56.812837 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.812809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:56.814816 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.814794 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ww8s\"/\"kube-root-ca.crt\"" Jul 24 16:41:56.814979 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.814843 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ww8s\"/\"openshift-service-ca.crt\"" Jul 24 16:41:56.815438 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.815416 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ww8s/must-gather-lndkr"] Jul 24 16:41:56.947750 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.947712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a56cfaa-7081-4c90-a5f6-106b7ff67946-must-gather-output\") pod \"must-gather-lndkr\" (UID: \"4a56cfaa-7081-4c90-a5f6-106b7ff67946\") " pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:56.947971 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:56.947839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsq8\" (UniqueName: \"kubernetes.io/projected/4a56cfaa-7081-4c90-a5f6-106b7ff67946-kube-api-access-rvsq8\") pod \"must-gather-lndkr\" (UID: \"4a56cfaa-7081-4c90-a5f6-106b7ff67946\") " pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:57.048925 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.048888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a56cfaa-7081-4c90-a5f6-106b7ff67946-must-gather-output\") pod \"must-gather-lndkr\" (UID: \"4a56cfaa-7081-4c90-a5f6-106b7ff67946\") " pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:57.049116 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.049019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsq8\" (UniqueName: \"kubernetes.io/projected/4a56cfaa-7081-4c90-a5f6-106b7ff67946-kube-api-access-rvsq8\") pod \"must-gather-lndkr\" (UID: \"4a56cfaa-7081-4c90-a5f6-106b7ff67946\") " pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:57.049219 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.049199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a56cfaa-7081-4c90-a5f6-106b7ff67946-must-gather-output\") pod \"must-gather-lndkr\" (UID: \"4a56cfaa-7081-4c90-a5f6-106b7ff67946\") " pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:57.056768 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.056745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsq8\" (UniqueName: \"kubernetes.io/projected/4a56cfaa-7081-4c90-a5f6-106b7ff67946-kube-api-access-rvsq8\") pod \"must-gather-lndkr\" (UID: \"4a56cfaa-7081-4c90-a5f6-106b7ff67946\") " pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:57.123647 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.123546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ww8s/must-gather-lndkr" Jul 24 16:41:57.261621 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.261597 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ww8s/must-gather-lndkr"] Jul 24 16:41:57.263844 ip-10-0-140-155 kubenswrapper[2580]: W0724 16:41:57.263815 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a56cfaa_7081_4c90_a5f6_106b7ff67946.slice/crio-1e348ceaa7f9c1b4c176b97e95b9fbc7b9da64dae3f155b043b7d9d84e7dfdbd WatchSource:0}: Error finding container 1e348ceaa7f9c1b4c176b97e95b9fbc7b9da64dae3f155b043b7d9d84e7dfdbd: Status 404 returned error can't find the container with id 1e348ceaa7f9c1b4c176b97e95b9fbc7b9da64dae3f155b043b7d9d84e7dfdbd Jul 24 16:41:57.265717 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.265703 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:41:57.659514 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:57.659484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ww8s/must-gather-lndkr" event={"ID":"4a56cfaa-7081-4c90-a5f6-106b7ff67946","Type":"ContainerStarted","Data":"1e348ceaa7f9c1b4c176b97e95b9fbc7b9da64dae3f155b043b7d9d84e7dfdbd"} Jul 24 16:41:58.668822 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:58.668780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ww8s/must-gather-lndkr" event={"ID":"4a56cfaa-7081-4c90-a5f6-106b7ff67946","Type":"ContainerStarted","Data":"d92c6ffd2e980df90e7e573320c208e13c8b25df912d5d783fe3b54a90e252a1"} Jul 24 16:41:58.669273 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:58.668838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ww8s/must-gather-lndkr" event={"ID":"4a56cfaa-7081-4c90-a5f6-106b7ff67946","Type":"ContainerStarted","Data":"db3b4b00074f158d7acb4795fdef8a799f2f8d5968b1f27b8cd53501c54239e4"} Jul 24 16:41:58.684087 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:58.684016 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6ww8s/must-gather-lndkr" podStartSLOduration=1.947287179 podStartE2EDuration="2.683995315s" totalImagesPullingTime="736.708136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:41:58.682590773 +0000 UTC m=+3287.204199316" watchObservedRunningTime="2026-07-24 16:41:58.683995315 +0000 UTC m=+3287.205603859" Jul 24 16:41:59.588954 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:59.588900 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-txr7v_b3c5e7b6-0692-4501-883c-29b33100cbd9/global-pull-secret-syncer/0.log" Jul 24 16:41:59.670443 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:59.670404 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hphrq_c6fe8d08-5deb-4923-b672-a15772b5065c/konnectivity-agent/0.log" Jul 24 16:41:59.778166 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:41:59.778137 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-155.ec2.internal_0202295370f109491358a25fd64440ce/haproxy/0.log" Jul 24 16:42:04.813959 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:04.813913 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-b7296_95bd019a-375b-44e2-84d9-0a306f8672a7/kuadrant-console-plugin/0.log" Jul 24 16:42:06.842547 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:06.842372 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d31bc6-c37b-4fcb-aa38-840ce2344737/alertmanager/0.log" Jul 24 16:42:06.865701 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:06.865681 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d31bc6-c37b-4fcb-aa38-840ce2344737/config-reloader/0.log" Jul 24 16:42:06.888899 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:06.888829 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d31bc6-c37b-4fcb-aa38-840ce2344737/kube-rbac-proxy-web/0.log" Jul 24 16:42:06.912134 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:06.912095 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d31bc6-c37b-4fcb-aa38-840ce2344737/kube-rbac-proxy/0.log" Jul 24 16:42:06.945998 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:06.945969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d31bc6-c37b-4fcb-aa38-840ce2344737/kube-rbac-proxy-metric/0.log" Jul 24 16:42:06.965199 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:06.965116 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d31bc6-c37b-4fcb-aa38-840ce2344737/prom-label-proxy/0.log" Jul 24 16:42:06.988996 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:06.988920 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3d31bc6-c37b-4fcb-aa38-840ce2344737/init-config-reloader/0.log" Jul 24 16:42:07.143500 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.143406 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5fc4d6979c-j5tfg_cc2ec951-5b31-401a-865e-6dc40fbff02b/metrics-server/0.log" Jul 24 16:42:07.364247 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.364219 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sczkq_85745cba-b8f4-4928-a9c6-65cc9ae288ac/node-exporter/0.log" Jul 24 16:42:07.386323 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.386298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sczkq_85745cba-b8f4-4928-a9c6-65cc9ae288ac/kube-rbac-proxy/0.log" Jul 24 16:42:07.408842 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.408764 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sczkq_85745cba-b8f4-4928-a9c6-65cc9ae288ac/init-textfile/0.log" Jul 24 16:42:07.727488 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.727417 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-jc2hp_288a192f-c4e8-4f57-b724-2787cee4b953/prometheus-operator-admission-webhook/0.log" Jul 24 16:42:07.764696 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.764658 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9cc584dff-957kg_c3e5cc0c-3eaa-4418-afd3-82f861e55003/telemeter-client/0.log" Jul 24 16:42:07.785795 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.785769 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9cc584dff-957kg_c3e5cc0c-3eaa-4418-afd3-82f861e55003/reload/0.log" Jul 24 16:42:07.808415 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.808368 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9cc584dff-957kg_c3e5cc0c-3eaa-4418-afd3-82f861e55003/kube-rbac-proxy/0.log" Jul 24 16:42:07.850520 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.850493 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cb95767d6-jnzr9_301f70fa-048d-48e6-b818-bb9f3f2a6243/thanos-query/0.log" Jul 24 16:42:07.869900 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.869865 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cb95767d6-jnzr9_301f70fa-048d-48e6-b818-bb9f3f2a6243/kube-rbac-proxy-web/0.log" Jul 24 16:42:07.892243 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.892216 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cb95767d6-jnzr9_301f70fa-048d-48e6-b818-bb9f3f2a6243/kube-rbac-proxy/0.log" Jul 24 16:42:07.920289 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.920264 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cb95767d6-jnzr9_301f70fa-048d-48e6-b818-bb9f3f2a6243/prom-label-proxy/0.log" Jul 24 16:42:07.945011 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.944981 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cb95767d6-jnzr9_301f70fa-048d-48e6-b818-bb9f3f2a6243/kube-rbac-proxy-rules/0.log" Jul 24 16:42:07.976176 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:07.976112 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cb95767d6-jnzr9_301f70fa-048d-48e6-b818-bb9f3f2a6243/kube-rbac-proxy-metrics/0.log" Jul 24 16:42:09.595833 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.595788 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l"] Jul 24 16:42:09.621960 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.621307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l"] Jul 24 16:42:09.621960 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.621435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.623902 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.623875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6ww8s\"/\"default-dockercfg-t8xkf\"" Jul 24 16:42:09.782237 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.782207 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:42:09.791584 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.791553 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/2.log" Jul 24 16:42:09.794780 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.794755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-podres\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.794905 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.794819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-proc\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.795029 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.794913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-sys\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.795077 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.795050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkc5z\" (UniqueName: \"kubernetes.io/projected/ae5582bc-6c35-4323-90ff-45ef0e06241f-kube-api-access-gkc5z\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.795124 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.795104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-lib-modules\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896609 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-proc\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896609 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-sys\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896848 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkc5z\" (UniqueName: \"kubernetes.io/projected/ae5582bc-6c35-4323-90ff-45ef0e06241f-kube-api-access-gkc5z\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896848 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-proc\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896848 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-lib-modules\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896848 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-sys\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896848 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-podres\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896848 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-lib-modules\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.896848 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.896786 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae5582bc-6c35-4323-90ff-45ef0e06241f-podres\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.904920 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.904878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkc5z\" (UniqueName: \"kubernetes.io/projected/ae5582bc-6c35-4323-90ff-45ef0e06241f-kube-api-access-gkc5z\") pod \"perf-node-gather-daemonset-ztg8l\" (UID: \"ae5582bc-6c35-4323-90ff-45ef0e06241f\") " pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:09.943478 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:09.943207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:10.116986 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:10.116952 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l"] Jul 24 16:42:10.763324 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:10.763286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" event={"ID":"ae5582bc-6c35-4323-90ff-45ef0e06241f","Type":"ContainerStarted","Data":"345f19bcf8a767145bd6736e28d7b6ba69557b3a641f4325bb11f2eff8bfd98c"} Jul 24 16:42:10.763743 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:10.763351 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" event={"ID":"ae5582bc-6c35-4323-90ff-45ef0e06241f","Type":"ContainerStarted","Data":"842c27ba03505aefae4edecf668ba6e47448511a4c85fb58820e92660867c91d"} Jul 24 16:42:10.763743 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:10.763499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:10.780439 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:10.780386 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" podStartSLOduration=1.7803636219999999 podStartE2EDuration="1.780363622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:42:10.778181427 +0000 UTC m=+3299.299789989" watchObservedRunningTime="2026-07-24 16:42:10.780363622 +0000 UTC m=+3299.301972166" Jul 24 16:42:11.002858 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:11.002827 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-flr6s_aa78a585-835c-49d6-85b5-caeae5da578c/volume-data-source-validator/0.log" Jul 24 16:42:12.041219 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:12.041193 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fw7z6_2f27ac16-5144-4a65-9a33-791650b37e8c/dns/0.log" Jul 24 16:42:12.061927 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:12.061906 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fw7z6_2f27ac16-5144-4a65-9a33-791650b37e8c/kube-rbac-proxy/0.log" Jul 24 16:42:12.180019 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:12.179984 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xmbr7_646c947d-c586-4522-9712-0204b10c4992/dns-node-resolver/0.log" Jul 24 16:42:12.255958 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:12.254617 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:42:12.267677 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:12.267652 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vzvhl_f014c88e-fe9e-4ed5-9579-9c5f721bc6ff/console-operator/1.log" Jul 24 16:42:12.722598 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:12.722499 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cf7574fb5-697wp_1cf83508-bfa7-467a-b016-f12406c82769/registry/0.log" Jul 24 16:42:12.744663 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:12.744639 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gfbz8_3b687088-c49c-4cdb-a4bd-d0099456e00d/node-ca/0.log" Jul 24 16:42:13.882311 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:13.882278 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545786b4d4-v222p_f3cd4577-c073-49d2-ad94-70f3ace80f8e/kube-auth-proxy/0.log" Jul 24 16:42:14.003771 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:14.003736 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9czsr_aba87454-6133-46b3-a7f1-1f2053b04e98/payload-processing/0.log" Jul 24 16:42:14.823067 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:14.823040 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srv29_8e0c010d-8dda-4460-89d2-4fcf81149a91/serve-healthcheck-canary/0.log" Jul 24 16:42:15.370181 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:15.370155 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvvwq_99e0f12c-caff-4e63-a132-ed8cea3448b9/kube-rbac-proxy/0.log" Jul 24 16:42:15.391766 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:15.391742 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvvwq_99e0f12c-caff-4e63-a132-ed8cea3448b9/exporter/0.log" Jul 24 16:42:15.413478 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:15.413455 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvvwq_99e0f12c-caff-4e63-a132-ed8cea3448b9/extractor/0.log" Jul 24 16:42:16.781052 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:16.781023 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6ww8s/perf-node-gather-daemonset-ztg8l" Jul 24 16:42:17.970857 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:17.970822 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cc959d7c4-vk82w_c754a2c8-8694-41c0-8b4f-bedd0be27e9f/manager/0.log" Jul 24 16:42:18.253245 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:18.253129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749946f7d7-swn4q_7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3/manager/0.log" Jul 24 16:42:18.274989 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:18.274961 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749946f7d7-swn4q_7fed1aca-09f7-4a1f-aaf6-ea679bf42fc3/copy-manifests/0.log" Jul 24 16:42:19.691725 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:19.691632 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-777c588886-r2r6f_cfc57b0f-baff-4518-a684-2d59590fb369/manager/0.log" Jul 24 16:42:20.317513 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:20.317482 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545786b4d4-v222p_f3cd4577-c073-49d2-ad94-70f3ace80f8e/kube-auth-proxy/0.log" Jul 24 16:42:20.417088 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:20.417054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9czsr_aba87454-6133-46b3-a7f1-1f2053b04e98/payload-processing/0.log" Jul 24 16:42:21.999285 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:21.999215 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545786b4d4-v222p_f3cd4577-c073-49d2-ad94-70f3ace80f8e/kube-auth-proxy/0.log" Jul 24 16:42:22.094646 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:22.094618 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9czsr_aba87454-6133-46b3-a7f1-1f2053b04e98/payload-processing/0.log" Jul 24 16:42:24.988032 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:24.988000 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq_563706c0-a505-4373-b987-923b16eb34da/kube-storage-version-migrator-operator/1.log" Jul 24 16:42:24.988839 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:24.988819 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-xc5gq_563706c0-a505-4373-b987-923b16eb34da/kube-storage-version-migrator-operator/0.log" Jul 24 16:42:25.997830 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:25.997801 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mk9f_e0add84e-ed07-4e3e-99ac-29a994e6b55e/kube-multus-additional-cni-plugins/0.log" Jul 24 16:42:26.017634 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.017605 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mk9f_e0add84e-ed07-4e3e-99ac-29a994e6b55e/egress-router-binary-copy/0.log" Jul 24 16:42:26.037996 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.037972 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mk9f_e0add84e-ed07-4e3e-99ac-29a994e6b55e/cni-plugins/0.log" Jul 24 16:42:26.058382 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.058352 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mk9f_e0add84e-ed07-4e3e-99ac-29a994e6b55e/bond-cni-plugin/0.log" Jul 24 16:42:26.078965 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.078925 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mk9f_e0add84e-ed07-4e3e-99ac-29a994e6b55e/routeoverride-cni/0.log" Jul 24 16:42:26.099740 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.099707 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mk9f_e0add84e-ed07-4e3e-99ac-29a994e6b55e/whereabouts-cni-bincopy/0.log" Jul 24 16:42:26.120028 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.120006 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mk9f_e0add84e-ed07-4e3e-99ac-29a994e6b55e/whereabouts-cni/0.log" Jul 24 16:42:26.454737 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.454707 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qswst_b04e560a-d00a-46d8-82ee-e9ae357824ee/kube-multus/0.log" Jul 24 16:42:26.508413 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.508388 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ngh44_1837b9cd-372b-4e0e-ab2a-ff6013c26aea/network-metrics-daemon/0.log" Jul 24 16:42:26.526611 ip-10-0-140-155 kubenswrapper[2580]: I0724 16:42:26.526585 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ngh44_1837b9cd-372b-4e0e-ab2a-ff6013c26aea/kube-rbac-proxy/0.log"