Jul 29 15:44:29.121072 ip-10-0-130-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 15:44:29.121085 ip-10-0-130-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 15:44:29.121096 ip-10-0-130-132 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 15:44:29.121442 ip-10-0-130-132 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 15:44:39.294746 ip-10-0-130-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 15:44:39.294763 ip-10-0-130-132 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4aa2adc0dda34e6081df4f0f40d8ae78 -- Jul 29 15:46:29.343877 ip-10-0-130-132 systemd[1]: Starting Kubernetes Kubelet... Jul 29 15:46:29.730283 ip-10-0-130-132 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 15:46:29.730283 ip-10-0-130-132 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 15:46:29.730283 ip-10-0-130-132 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 15:46:29.730283 ip-10-0-130-132 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 15:46:29.730283 ip-10-0-130-132 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 15:46:29.732289 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.732161 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 15:46:29.738453 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738427 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:29.738453 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738450 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:29.738453 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738454 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738457 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738460 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738464 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738467 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738470 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738472 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738476 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738479 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738481 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738484 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738487 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738490 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738493 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738511 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738514 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738517 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738520 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738523 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:29.738555 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738525 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738528 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738531 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738534 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738536 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738539 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738542 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738544 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738547 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738550 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738552 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738555 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738557 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738560 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738563 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738566 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738569 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738572 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738576 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738590 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:29.739032 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738594 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738597 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738600 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738603 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738606 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738609 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738611 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738614 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738617 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738619 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738622 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738625 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738628 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738630 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738638 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738640 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738644 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738646 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738649 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738651 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:29.739519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738654 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738657 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738659 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738662 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738666 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738669 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738672 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738675 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738677 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738680 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738684 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738686 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738689 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738692 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738696 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738700 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738703 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738706 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738709 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:29.740062 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738712 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738714 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738717 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738719 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738722 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738724 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738727 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738731 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738734 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738737 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738739 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738742 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738745 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738748 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738750 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738753 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738755 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738757 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.738760 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739550 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:29.740523 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739558 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739563 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739566 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739570 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739577 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739597 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739600 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739604 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739608 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739611 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739613 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739616 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739619 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739622 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739626 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739629 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739632 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739634 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739637 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:29.741052 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739641 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739644 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739646 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739649 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739651 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739653 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739656 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739658 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739661 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739663 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739666 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739668 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739670 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739673 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739675 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739678 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739680 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739684 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739688 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739692 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739696 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:29.741522 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739700 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739723 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739726 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739729 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739731 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739734 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739736 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739738 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739741 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739743 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739746 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739750 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739753 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739755 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739758 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739761 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739763 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739766 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739770 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739773 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:29.742057 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739775 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739778 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739781 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739784 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739787 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739790 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739792 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739795 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739799 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739801 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739805 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739807 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739810 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739812 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739815 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739818 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739820 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739823 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739825 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:29.742544 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739828 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739830 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739832 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739835 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739838 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739841 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739843 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739846 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739849 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739852 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739854 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739857 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739859 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739862 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739864 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739866 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739869 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739872 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.739874 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:29.743031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740287 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740294 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740298 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740301 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740305 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740308 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740311 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740314 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740316 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740319 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740321 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740324 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740327 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740331 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740335 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740338 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740341 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740344 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740347 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:29.743503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740349 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740352 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740355 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740357 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740360 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740362 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740365 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740368 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740370 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740372 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740375 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740377 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740379 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740382 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740385 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740388 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740390 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740393 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740396 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740399 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:29.744029 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740403 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740406 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740409 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740412 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740415 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740417 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740420 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740422 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740425 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740429 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740431 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740434 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740436 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740439 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740441 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740444 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740447 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740450 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740453 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:29.744519 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740455 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740458 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740460 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740463 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740465 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740468 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740470 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740473 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740476 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740479 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740481 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740484 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740487 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740490 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740492 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740495 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740497 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740500 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740502 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740505 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:29.745092 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740507 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740510 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740514 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740517 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740519 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740521 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740524 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740527 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740529 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740532 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740535 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740538 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740540 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740542 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740545 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740547 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740550 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740552 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740556 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740559 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:29.745644 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.740561 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740657 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740665 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740673 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740677 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740682 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740686 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740690 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740704 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740707 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740710 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740714 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740717 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740720 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740725 2555 flags.go:64] FLAG: --cgroup-root="" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740728 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740731 2555 flags.go:64] FLAG: --client-ca-file="" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740733 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740736 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740741 2555 flags.go:64] FLAG: --cluster-domain="" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740744 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740747 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740750 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740753 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 15:46:29.746141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740757 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740760 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740764 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740767 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740770 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740773 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740777 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740780 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740783 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740788 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740791 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740794 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740797 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740800 2555 flags.go:64] FLAG: --enable-server="true" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740804 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740809 2555 flags.go:64] FLAG: --event-burst="100" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740813 2555 flags.go:64] FLAG: --event-qps="50" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740816 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740818 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740822 2555 flags.go:64] FLAG: --eviction-hard="" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740826 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740829 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740834 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740837 2555 flags.go:64] FLAG: --eviction-soft="" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740840 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 15:46:29.746731 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740842 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740845 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740848 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740851 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740854 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740857 2555 flags.go:64] FLAG: --feature-gates="" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740861 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740864 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740868 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740871 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740874 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740876 2555 flags.go:64] FLAG: --help="false" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740879 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-130-132.ec2.internal" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740884 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740887 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740890 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740893 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740896 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740899 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740902 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740905 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740908 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740911 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740914 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 15:46:29.747430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740917 2555 flags.go:64] FLAG: --kube-reserved="" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740920 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740923 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740926 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740929 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740933 2555 flags.go:64] FLAG: --lock-file="" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740936 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740938 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740942 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740947 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740950 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740953 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740955 2555 flags.go:64] FLAG: --logging-format="text" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740958 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740962 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740965 2555 flags.go:64] FLAG: --manifest-url="" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740967 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740972 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740975 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740979 2555 flags.go:64] FLAG: --max-pods="110" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740983 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740987 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740990 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740993 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740996 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 15:46:29.748072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.740999 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741002 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741011 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741014 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741017 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741020 2555 flags.go:64] FLAG: --pod-cidr="" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741023 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741029 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741032 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741036 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741039 2555 flags.go:64] FLAG: --port="10250" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741042 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741045 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b1475ef5bc2c1b67" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741049 2555 flags.go:64] FLAG: --qos-reserved="" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741052 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741055 2555 flags.go:64] FLAG: --register-node="true" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741058 2555 flags.go:64] FLAG: --register-with-taints="" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741063 2555 flags.go:64] FLAG: --registry-burst="10" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741066 2555 flags.go:64] FLAG: --registry-qps="5" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741069 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741072 2555 flags.go:64] FLAG: --reserved-memory="" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741076 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741079 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741082 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741085 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 15:46:29.748744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741088 2555 flags.go:64] FLAG: --runonce="false" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741091 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741094 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741097 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741100 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741103 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741107 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741110 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741113 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741116 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741119 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741121 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741124 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741127 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741131 2555 flags.go:64] FLAG: --system-cgroups="" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741134 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741139 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741142 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741145 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741149 2555 flags.go:64] FLAG: --tls-min-version="" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741154 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741157 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741159 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741162 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741165 2555 flags.go:64] FLAG: --v="2" Jul 29 15:46:29.749370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741169 2555 flags.go:64] FLAG: --version="false" Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741173 2555 flags.go:64] FLAG: --vmodule="" Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741177 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741180 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741303 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741307 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741310 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741312 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741315 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741318 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741321 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741324 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741327 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741329 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741332 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741335 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741338 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741340 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741343 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741345 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:29.749999 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741348 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741350 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741356 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741358 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741361 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741363 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741365 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741370 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741373 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741375 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741377 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741380 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741383 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741386 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741389 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741391 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741394 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741397 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741400 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741402 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:29.750493 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741404 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741407 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741409 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741412 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741414 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741419 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741422 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741425 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741428 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741431 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741434 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741436 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741439 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741442 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741444 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741447 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741450 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741453 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741455 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:29.751036 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741460 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741463 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741465 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741467 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741470 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741472 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741475 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741478 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741480 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741483 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741485 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741488 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741490 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741492 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741495 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741498 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741500 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741502 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741505 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741507 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:29.751503 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741510 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741512 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741514 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741518 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741522 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741525 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741529 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741532 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741535 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741538 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741541 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741544 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741547 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741550 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741553 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741555 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741557 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741560 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741564 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:29.752031 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741566 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741569 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741571 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741574 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.741576 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.741596 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.748420 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.748438 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.748459 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.748465 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748527 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748532 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748536 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748539 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748542 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:29.752490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748545 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748548 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748551 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748553 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748556 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748559 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748561 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748564 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748566 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748569 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748571 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748574 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748577 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748598 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748603 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748606 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748609 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748612 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748615 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:29.752897 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748618 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748620 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748623 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748625 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748628 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748632 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748635 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748638 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748641 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748644 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748646 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748649 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748652 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748654 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748657 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748659 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748662 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748664 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748669 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:29.753386 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748673 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748676 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748679 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748682 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748685 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748687 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748690 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748692 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748695 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748698 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748700 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748703 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748706 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748708 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748711 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748714 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748717 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748719 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748724 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748728 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:29.753868 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748731 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748734 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748737 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748740 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748743 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748746 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748749 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748751 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748770 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748774 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748777 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748780 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748783 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748785 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748788 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748791 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748793 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748796 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748798 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748801 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748803 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:29.754398 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748806 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748809 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748811 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748814 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748816 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748819 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748822 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748824 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748827 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748830 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748833 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748836 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748839 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748842 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748844 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:29.754933 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.748849 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748977 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748983 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748986 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748988 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748991 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748995 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.748999 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749002 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749005 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749008 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749010 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749013 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749015 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749018 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749021 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749024 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749026 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749028 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:29.755316 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749031 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749033 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749036 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749040 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749043 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749046 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749049 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749052 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749055 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749058 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749061 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749063 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749066 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749069 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749071 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749074 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749077 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749079 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749082 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:29.755920 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749084 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749087 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749089 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749092 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749095 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749097 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749100 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749102 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749105 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749108 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749110 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749113 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749116 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749118 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749121 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749123 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749125 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749128 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749131 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749133 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:29.756446 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749136 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749139 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749141 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749144 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749147 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749149 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749152 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749154 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749157 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749160 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749163 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749165 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749168 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749170 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749173 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749175 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749178 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749181 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749183 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:29.757060 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749186 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749188 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749191 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749193 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749196 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749198 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749201 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749203 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749206 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749208 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749211 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749215 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749217 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749220 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749223 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749225 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749228 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749230 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749233 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749236 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:29.757693 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749238 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749241 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:29.749243 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.749248 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.749394 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.751612 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.753071 2555 server.go:1019] "Starting client certificate rotation" Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.753171 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.753202 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 15:46:29.758189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.755453 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 15:46:29.759308 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.759292 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 15:46:29.769438 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.769415 2555 log.go:25] "Validated CRI v1 runtime API" Jul 29 15:46:29.772468 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.772450 2555 log.go:25] "Validated CRI v1 image API" Jul 29 15:46:29.773964 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.773946 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 15:46:29.779264 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.779239 2555 fs.go:135] Filesystem UUIDs: map[14c3d93b-0343-435d-b9a1-7279b3ad51a0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dd17a5c1-b7da-4f53-835b-853516cda46c:/dev/nvme0n1p3] Jul 29 15:46:29.779350 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.779261 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 15:46:29.784789 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.784766 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 15:46:29.784841 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.784675 2555 manager.go:216] Machine: {Timestamp:2026-07-29 15:46:29.783501631 +0000 UTC m=+0.324403091 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094978 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20c4a5da27e3458b11567d1c6295de SystemUUID:ec20c4a5-da27-e345-8b11-567d1c6295de BootID:4aa2adc0-dda3-4e60-81df-4f0f40d8ae78 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:21:d7:6b:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:21:d7:6b:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:63:e1:9a:78:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 15:46:29.784841 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.784826 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 15:46:29.784947 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.784934 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 15:46:29.785980 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.785952 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 15:46:29.786139 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.785982 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 15:46:29.787344 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.787330 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 15:46:29.787382 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.787348 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 15:46:29.787382 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.787356 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 15:46:29.787923 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.787912 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 15:46:29.788827 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.788815 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 15:46:29.790699 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.790685 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 29 15:46:29.790836 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.790826 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 15:46:29.790885 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.790865 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 29 15:46:29.790885 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.790877 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 15:46:29.790942 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.790890 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 15:46:29.790942 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.790902 2555 kubelet.go:398] "Adding apiserver pod source" Jul 29 15:46:29.790942 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.790917 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 15:46:29.791445 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.791432 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 15:46:29.791602 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.791455 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 15:46:29.798179 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.798157 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 15:46:29.800089 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.800069 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 15:46:29.800194 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.800112 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 15:46:29.801607 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801574 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801612 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801619 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801625 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801634 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801639 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801649 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801654 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801662 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801667 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801682 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 15:46:29.801687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.801687 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 15:46:29.802447 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.802426 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 15:46:29.802658 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.802639 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 15:46:29.802925 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.802913 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 15:46:29.802957 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.802926 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 15:46:29.805657 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.805640 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 15:46:29.806082 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.806071 2555 server.go:1268] "Started kubelet" Jul 29 15:46:29.806169 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.806141 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 15:46:29.806271 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.806206 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 15:46:29.806311 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.806303 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 15:46:29.807080 ip-10-0-130-132 systemd[1]: Started Kubernetes Kubelet. Jul 29 15:46:29.807886 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.807872 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 15:46:29.808483 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.808465 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 29 15:46:29.812156 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.812124 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 15:46:29.815391 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.815374 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 15:46:29.815391 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.815380 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 15:46:29.815527 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.815495 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:29.816253 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816104 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 15:46:29.816253 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816108 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 15:46:29.816253 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816136 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 15:46:29.816434 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816260 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 29 15:46:29.816434 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816270 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 29 15:46:29.816434 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816382 2555 factory.go:55] Registering systemd factory Jul 29 15:46:29.816434 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816398 2555 factory.go:223] Registration of the systemd container factory successfully Jul 29 15:46:29.816742 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816723 2555 factory.go:153] Registering CRI-O factory Jul 29 15:46:29.816818 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816745 2555 factory.go:223] Registration of the crio container factory successfully Jul 29 15:46:29.816872 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816818 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 15:46:29.816872 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816871 2555 factory.go:103] Registering Raw factory Jul 29 15:46:29.816965 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.816887 2555 manager.go:1196] Started watching for new ooms in manager Jul 29 15:46:29.816965 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.815748 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-132.ec2.internal.18c6cd42f7b045bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-132.ec2.internal,UID:ip-10-0-130-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-132.ec2.internal,},FirstTimestamp:2026-07-29 15:46:29.806040509 +0000 UTC m=+0.346941968,LastTimestamp:2026-07-29 15:46:29.806040509 +0000 UTC m=+0.346941968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-132.ec2.internal,}" Jul 29 15:46:29.817481 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.817430 2555 manager.go:349] Starting recovery of all containers Jul 29 15:46:29.820636 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.820605 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 15:46:29.820746 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.820704 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 15:46:29.832404 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.832379 2555 manager.go:354] Recovery completed Jul 29 15:46:29.837403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.837386 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:29.839593 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.839563 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:29.839645 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.839611 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:29.839645 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.839625 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:29.840230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.840215 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 15:46:29.840230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.840228 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 15:46:29.840230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.840230 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:29.840384 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.840245 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 29 15:46:29.841815 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.841794 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:29.841909 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.841827 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:29.841909 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.841842 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:29.842092 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.841940 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-132.ec2.internal.18c6cd42f9b0463c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-132.ec2.internal,UID:ip-10-0-130-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-132.ec2.internal,},FirstTimestamp:2026-07-29 15:46:29.839595068 +0000 UTC m=+0.380496533,LastTimestamp:2026-07-29 15:46:29.839595068 +0000 UTC m=+0.380496533,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-132.ec2.internal,}" Jul 29 15:46:29.842167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.842099 2555 policy_none.go:49] "None policy: Start" Jul 29 15:46:29.842167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.842119 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 15:46:29.842167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.842133 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 15:46:29.843130 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.843118 2555 policy_none.go:47] "Start" Jul 29 15:46:29.850384 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.850362 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwg9w" Jul 29 15:46:29.852536 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.852438 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-132.ec2.internal.18c6cd42f9b09b87 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-132.ec2.internal,UID:ip-10-0-130-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-132.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-132.ec2.internal,},FirstTimestamp:2026-07-29 15:46:29.839616903 +0000 UTC m=+0.380518366,LastTimestamp:2026-07-29 15:46:29.839616903 +0000 UTC m=+0.380518366,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-132.ec2.internal,}" Jul 29 15:46:29.857833 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.857815 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwg9w" Jul 29 15:46:29.884568 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.884365 2555 manager.go:337] "Starting Device Plugin manager" Jul 29 15:46:29.884735 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.884587 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 15:46:29.884735 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.884603 2555 server.go:87] "Starting device plugin registration server" Jul 29 15:46:29.884953 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.884938 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 15:46:29.885040 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.884955 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 15:46:29.885103 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.885051 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 15:46:29.885163 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.885150 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 15:46:29.885163 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.885162 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 15:46:29.885996 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.885971 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 15:46:29.886082 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.886037 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:29.916445 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.916401 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 15:46:29.918217 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.917641 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 15:46:29.918217 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.917694 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 15:46:29.918217 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.917733 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 15:46:29.918217 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.917788 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 15:46:29.920252 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.920225 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:29.988246 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.988149 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:29.989242 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.989224 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:29.989340 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.989258 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:29.989340 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.989273 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:29.989340 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.989303 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-132.ec2.internal" Jul 29 15:46:29.998938 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:29.998915 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-132.ec2.internal" Jul 29 15:46:29.998995 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:29.998943 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-132.ec2.internal\": node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.018690 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.018655 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"] Jul 29 15:46:30.018787 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.018736 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:30.020919 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.020899 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:30.021014 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.020931 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:30.021014 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.020942 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:30.022485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.022470 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:30.022610 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.022580 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.022653 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.022630 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:30.023910 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.023887 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:30.023995 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.023923 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:30.023995 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.023933 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:30.023995 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.023887 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:30.024100 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.024013 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:30.024100 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.024027 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:30.025239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.025222 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.025338 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.025246 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:30.026093 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.026077 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:30.026182 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.026107 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:30.026182 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.026119 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:30.042370 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.042345 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-132.ec2.internal\" not found" node="ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.046668 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.046633 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-132.ec2.internal\" not found" node="ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.054341 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.054323 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.117695 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.117659 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f09a92c41c6e0b2866952bb93aa6998f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"f09a92c41c6e0b2866952bb93aa6998f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.117824 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.117703 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a573c2ec0c2e9c2aaedfbe717cd64a75-config\") pod \"kube-apiserver-proxy-ip-10-0-130-132.ec2.internal\" (UID: \"a573c2ec0c2e9c2aaedfbe717cd64a75\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.117824 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.117725 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f09a92c41c6e0b2866952bb93aa6998f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"f09a92c41c6e0b2866952bb93aa6998f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.156668 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.156628 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.219673 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.219639 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a573c2ec0c2e9c2aaedfbe717cd64a75-config\") pod \"kube-apiserver-proxy-ip-10-0-130-132.ec2.internal\" (UID: \"a573c2ec0c2e9c2aaedfbe717cd64a75\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.219673 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.219678 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f09a92c41c6e0b2866952bb93aa6998f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"f09a92c41c6e0b2866952bb93aa6998f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.219775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.219697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f09a92c41c6e0b2866952bb93aa6998f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"f09a92c41c6e0b2866952bb93aa6998f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.219775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.219733 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f09a92c41c6e0b2866952bb93aa6998f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"f09a92c41c6e0b2866952bb93aa6998f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.219775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.219746 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a573c2ec0c2e9c2aaedfbe717cd64a75-config\") pod \"kube-apiserver-proxy-ip-10-0-130-132.ec2.internal\" (UID: \"a573c2ec0c2e9c2aaedfbe717cd64a75\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.219775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.219770 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f09a92c41c6e0b2866952bb93aa6998f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"f09a92c41c6e0b2866952bb93aa6998f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.257696 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.257626 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.345946 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.345911 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.350868 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.350350 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.358030 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.358001 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.460889 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.460835 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.564197 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.564116 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.629084 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.629047 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:30.665037 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.665001 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found" Jul 29 15:46:30.744989 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.744958 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:30.753011 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.752981 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 15:46:30.753195 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.753130 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 15:46:30.753195 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.753177 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 15:46:30.753287 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.753183 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 15:46:30.791671 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.791631 2555 apiserver.go:52] "Watching apiserver" Jul 29 15:46:30.811627 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.811588 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 15:46:30.812192 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.812173 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:30.812816 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.812788 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-bp8mc","openshift-network-diagnostics/network-check-target-gqd46","openshift-network-operator/iptables-alerter-vqlfg","openshift-ovn-kubernetes/ovnkube-node-nqdlk","kube-system/global-pull-secret-syncer-ptml9","kube-system/konnectivity-agent-skhtl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv","openshift-cluster-node-tuning-operator/tuned-6xb6w","openshift-dns/node-resolver-m7jv2","openshift-image-registry/node-ca-6mvd9","openshift-multus/multus-additional-cni-plugins-8pzdz","openshift-multus/network-metrics-daemon-29fg9"] Jul 29 15:46:30.814604 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.814562 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.815760 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.815569 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.815760 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.815592 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 15:46:30.816113 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.816088 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:30.816208 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.816170 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:30.817053 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.817034 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8dnqm\"" Jul 29 15:46:30.817205 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.817060 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.817589 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.817541 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 15:46:30.817837 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.817816 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 15:46:30.817943 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.817912 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 15:46:30.818524 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.818505 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 15:46:30.818524 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.818520 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.820060 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.820020 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:30.820060 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.820046 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 15:46:30.820300 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.820096 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:30.820300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.820022 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:46:30.820300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.820144 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dsstd\"" Jul 29 15:46:30.820486 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.820358 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 15:46:30.821503 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.821187 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 15:46:30.821503 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.821407 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:30.822018 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.821993 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 15:46:30.822018 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.822016 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 15:46:30.822171 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.822007 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 15:46:30.822171 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.822034 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 15:46:30.822282 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.822247 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 15:46:30.822432 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.822415 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-blxnj\"" Jul 29 15:46:30.822874 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.822856 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823289 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-kubelet\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823342 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-system-cni-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823375 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-cni-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-os-release\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823442 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-netns\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-cni-multus\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823499 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-hostroot\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823531 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/063034e5-79e7-4ffe-8748-75e0452a44e1-cni-binary-copy\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823565 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-k8s-cni-cncf-io\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823760 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-daemon-config\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.823786 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxmk\" (UniqueName: \"kubernetes.io/projected/063034e5-79e7-4ffe-8748-75e0452a44e1-kube-api-access-bfxmk\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824000 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824055 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/224b987f-fb05-4ebf-ac43-938f2f3b9340-iptables-alerter-script\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824092 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-socket-dir-parent\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.824123 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824120 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-conf-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.825013 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824151 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-multus-certs\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.825013 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824181 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/224b987f-fb05-4ebf-ac43-938f2f3b9340-host-slash\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.825013 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824214 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxgh\" (UniqueName: \"kubernetes.io/projected/224b987f-fb05-4ebf-ac43-938f2f3b9340-kube-api-access-wqxgh\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.825013 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-cnibin\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.825013 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824271 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-cni-bin\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.825013 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.824301 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-etc-kubernetes\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.826638 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.826614 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.827178 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.827157 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j7lrt\"" Jul 29 15:46:30.827279 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.827162 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 15:46:30.827547 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.827531 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 15:46:30.827627 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.827553 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 15:46:30.827627 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.827582 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 15:46:30.827771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.827718 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 15:46:30.827835 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.827726 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h2xh6\"" Jul 29 15:46:30.828035 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.828021 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:30.829235 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.829214 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:46:30.829405 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.829385 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q8279\"" Jul 29 15:46:30.829487 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.829387 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 15:46:30.829928 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.829908 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:30.831132 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.831114 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b5j8b\"" Jul 29 15:46:30.831209 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.831130 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 15:46:30.831406 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.831391 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 15:46:30.831546 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.831509 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.832546 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.832526 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 15:46:30.832681 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.832663 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 15:46:30.832810 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.832767 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6q7s6\"" Jul 29 15:46:30.832910 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.832888 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 15:46:30.832972 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.832955 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:30.833030 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.833013 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:30.833882 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.833861 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"] Jul 29 15:46:30.834122 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.834108 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 15:46:30.835070 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.835055 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 15:46:30.835217 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.835091 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 15:46:30.835217 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.835117 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" Jul 29 15:46:30.835317 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.835275 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6nln2\"" Jul 29 15:46:30.837469 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.837451 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 15:46:30.851508 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.851485 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 15:46:30.852658 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.852631 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"] Jul 29 15:46:30.863119 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.863069 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 15:41:30 +0000 UTC" deadline="2028-01-04 15:24:29.962271605 +0000 UTC" Jul 29 15:46:30.863119 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.863114 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12575h37m59.099160792s" Jul 29 15:46:30.865072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.865053 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sxhxr" Jul 29 15:46:30.875113 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.875088 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sxhxr" Jul 29 15:46:30.915077 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:30.915031 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda573c2ec0c2e9c2aaedfbe717cd64a75.slice/crio-166d61cc35bd104ba706a12ab62064d6a53a07b695696994957d87ac9646c24c WatchSource:0}: Error finding container 166d61cc35bd104ba706a12ab62064d6a53a07b695696994957d87ac9646c24c: Status 404 returned error can't find the container with id 166d61cc35bd104ba706a12ab62064d6a53a07b695696994957d87ac9646c24c Jul 29 15:46:30.915844 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:30.915820 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09a92c41c6e0b2866952bb93aa6998f.slice/crio-9325e9c1aec67ab610393d998ff34d29bb02cbda4d64351c245d642e0374fd6d WatchSource:0}: Error finding container 9325e9c1aec67ab610393d998ff34d29bb02cbda4d64351c245d642e0374fd6d: Status 404 returned error can't find the container with id 9325e9c1aec67ab610393d998ff34d29bb02cbda4d64351c245d642e0374fd6d Jul 29 15:46:30.917279 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.917255 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 15:46:30.921258 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.921233 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 15:46:30.921487 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.921441 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" event={"ID":"a573c2ec0c2e9c2aaedfbe717cd64a75","Type":"ContainerStarted","Data":"166d61cc35bd104ba706a12ab62064d6a53a07b695696994957d87ac9646c24c"} Jul 29 15:46:30.922876 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.922851 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" event={"ID":"f09a92c41c6e0b2866952bb93aa6998f","Type":"ContainerStarted","Data":"9325e9c1aec67ab610393d998ff34d29bb02cbda4d64351c245d642e0374fd6d"} Jul 29 15:46:30.924513 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924495 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxmk\" (UniqueName: \"kubernetes.io/projected/063034e5-79e7-4ffe-8748-75e0452a44e1-kube-api-access-bfxmk\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.924612 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924527 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-registration-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.924612 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924548 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-netns\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.924612 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924571 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/120f7d41-d050-48d7-acd8-c9779c7af52f-host\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:30.924771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924593 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/120f7d41-d050-48d7-acd8-c9779c7af52f-serviceca\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:30.924771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924641 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:30.924771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924672 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-node-log\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.924771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924697 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-env-overrides\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.924771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924687 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-netns\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.924771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924733 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-var-lib-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.925010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924778 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-log-socket\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.925010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924829 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-cni-netd\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.925010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924864 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-ovnkube-config\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.925010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924892 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-ovnkube-script-lib\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.925010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924919 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-systemd\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.925010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924944 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be264cf7-2ed3-4c99-8a85-264cacb645c7-hosts-file\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:30.925010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.924971 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be264cf7-2ed3-4c99-8a85-264cacb645c7-tmp-dir\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:30.925295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925015 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxgh\" (UniqueName: \"kubernetes.io/projected/224b987f-fb05-4ebf-ac43-938f2f3b9340-kube-api-access-wqxgh\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.925295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925069 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrm2\" (UniqueName: \"kubernetes.io/projected/69cffac7-1815-42f4-8bc7-ed8a8996b20f-kube-api-access-6zrm2\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:30.925295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925172 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69cf0189-a6a2-406d-991b-35b69c360c09-agent-certs\") pod \"konnectivity-agent-skhtl\" (UID: \"69cf0189-a6a2-406d-991b-35b69c360c09\") " pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:30.925295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925210 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-modprobe-d\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.925295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925237 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysctl-conf\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.925295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925268 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwzh\" (UniqueName: \"kubernetes.io/projected/be264cf7-2ed3-4c99-8a85-264cacb645c7-kube-api-access-lwwzh\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:30.925550 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925322 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xcdm\" (UniqueName: \"kubernetes.io/projected/8341f0c8-f991-4c19-a797-f7fce085608f-kube-api-access-4xcdm\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.925550 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925398 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-etc-kubernetes\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.925550 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925435 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khh5\" (UniqueName: \"kubernetes.io/projected/c7231bed-9e54-4bc2-8b99-c3053f2839b8-kube-api-access-6khh5\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.925550 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925462 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-lib-modules\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.925550 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925466 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-etc-kubernetes\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.925550 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925507 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.925550 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925561 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-cni-bin\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.925830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925577 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-socket-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.925830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925635 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-system-cni-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.925830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925659 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-os-release\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.925830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925700 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.925830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-sys\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.925830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925774 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/063034e5-79e7-4ffe-8748-75e0452a44e1-cni-binary-copy\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.925830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925752 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-system-cni-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925829 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-os-release\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.925982 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-device-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.926167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926035 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46p4\" (UniqueName: \"kubernetes.io/projected/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-kube-api-access-s46p4\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.926167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926066 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-cnibin\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.926167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926105 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-conf-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926135 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-systemd-units\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.926167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926161 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-run-netns\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926185 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926229 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-kubernetes\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926281 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-conf-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926270 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-tuned\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926335 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-cni-bin\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926363 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-etc-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926387 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-ovn\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926409 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f112c12-949a-414b-a960-8babbdbd28e9-ovn-node-metrics-cert\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926423 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-cni-bin\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926436 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.926485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926466 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/224b987f-fb05-4ebf-ac43-938f2f3b9340-iptables-alerter-script\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926589 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/063034e5-79e7-4ffe-8748-75e0452a44e1-cni-binary-copy\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-cni-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926653 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-cni-multus\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926728 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-cni-multus\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926775 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-hostroot\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926788 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-cni-dir\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926802 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-hostroot\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926804 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-run\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926840 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-k8s-cni-cncf-io\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926901 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-slash\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926913 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-k8s-cni-cncf-io\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926919 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-socket-dir-parent\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926939 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-multus-certs\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926956 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/224b987f-fb05-4ebf-ac43-938f2f3b9340-host-slash\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.926968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926977 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6d0e693-ccf2-4168-a63c-2f1428af95fe-dbus\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.926978 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-socket-dir-parent\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927026 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-run-multus-certs\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927032 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysctl-d\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927073 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/224b987f-fb05-4ebf-ac43-938f2f3b9340-host-slash\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927410 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-var-lib-kubelet\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927486 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysconfig\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927512 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-host\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927542 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-tmp\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:30.927767 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927644 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.928156 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927849 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/224b987f-fb05-4ebf-ac43-938f2f3b9340-iptables-alerter-script\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.928156 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927894 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:30.928156 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.927944 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.928335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928169 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-kubelet\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.928335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928179 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-host-var-lib-kubelet\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.928335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928221 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-system-cni-dir\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.928335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928254 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-os-release\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.928335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928304 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:30.928634 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928343 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-daemon-config\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.928634 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928397 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-systemd\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.928634 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928496 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ch9\" (UniqueName: \"kubernetes.io/projected/9f112c12-949a-414b-a960-8babbdbd28e9-kube-api-access-q4ch9\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.928634 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928536 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-sys-fs\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:30.928634 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928586 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkh2v\" (UniqueName: \"kubernetes.io/projected/120f7d41-d050-48d7-acd8-c9779c7af52f-kube-api-access-wkh2v\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:30.928634 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928612 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-kubelet\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.928960 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928708 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:30.928960 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928740 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69cf0189-a6a2-406d-991b-35b69c360c09-konnectivity-ca\") pod \"konnectivity-agent-skhtl\" (UID: \"69cf0189-a6a2-406d-991b-35b69c360c09\") " pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:30.928960 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928776 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-cnibin\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.928960 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928829 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6d0e693-ccf2-4168-a63c-2f1428af95fe-kubelet-config\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:30.928960 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928859 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:30.928960 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.928881 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/063034e5-79e7-4ffe-8748-75e0452a44e1-multus-daemon-config\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.929291 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.929062 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/063034e5-79e7-4ffe-8748-75e0452a44e1-cnibin\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.932448 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.932424 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 15:46:30.935755 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.935710 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxmk\" (UniqueName: \"kubernetes.io/projected/063034e5-79e7-4ffe-8748-75e0452a44e1-kube-api-access-bfxmk\") pod \"multus-bp8mc\" (UID: \"063034e5-79e7-4ffe-8748-75e0452a44e1\") " pod="openshift-multus/multus-bp8mc" Jul 29 15:46:30.935755 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:30.935727 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxgh\" (UniqueName: \"kubernetes.io/projected/224b987f-fb05-4ebf-ac43-938f2f3b9340-kube-api-access-wqxgh\") pod \"iptables-alerter-vqlfg\" (UID: \"224b987f-fb05-4ebf-ac43-938f2f3b9340\") " pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:30.937243 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.937227 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:30.937291 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.937249 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:30.937291 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.937260 2555 projected.go:196] Error preparing data for projected volume kube-api-access-2xjwl for pod openshift-network-diagnostics/network-check-target-gqd46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:30.937360 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:30.937326 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl podName:02e74657-8c54-477b-a184-eb2a35c8eff5 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:31.437301239 +0000 UTC m=+1.978202713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2xjwl" (UniqueName: "kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl") pod "network-check-target-gqd46" (UID: "02e74657-8c54-477b-a184-eb2a35c8eff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:31.029230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be264cf7-2ed3-4c99-8a85-264cacb645c7-tmp-dir\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:31.029230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029237 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrm2\" (UniqueName: \"kubernetes.io/projected/69cffac7-1815-42f4-8bc7-ed8a8996b20f-kube-api-access-6zrm2\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:31.029492 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029261 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69cf0189-a6a2-406d-991b-35b69c360c09-agent-certs\") pod \"konnectivity-agent-skhtl\" (UID: \"69cf0189-a6a2-406d-991b-35b69c360c09\") " pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:31.029492 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029285 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-modprobe-d\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029492 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029310 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysctl-conf\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029492 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029442 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-modprobe-d\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029492 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029450 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwzh\" (UniqueName: \"kubernetes.io/projected/be264cf7-2ed3-4c99-8a85-264cacb645c7-kube-api-access-lwwzh\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:31.029492 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029467 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysctl-conf\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xcdm\" (UniqueName: \"kubernetes.io/projected/8341f0c8-f991-4c19-a797-f7fce085608f-kube-api-access-4xcdm\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khh5\" (UniqueName: \"kubernetes.io/projected/c7231bed-9e54-4bc2-8b99-c3053f2839b8-kube-api-access-6khh5\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029583 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-lib-modules\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029629 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029649 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be264cf7-2ed3-4c99-8a85-264cacb645c7-tmp-dir\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029660 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-cni-bin\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029688 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-socket-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029717 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029728 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-cni-bin\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-sys\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-device-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029806 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s46p4\" (UniqueName: \"kubernetes.io/projected/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-kube-api-access-s46p4\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029815 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-lib-modules\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029828 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-socket-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029832 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-cnibin\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.029891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029883 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-cnibin\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029884 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-systemd-units\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029916 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-systemd-units\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029902 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-sys\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-device-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-run-netns\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.029985 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030040 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-run-netns\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030047 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-kubernetes\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030080 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030082 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-tuned\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030106 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-kubernetes\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030127 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-etc-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030156 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-ovn\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030174 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-etc-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030182 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f112c12-949a-414b-a960-8babbdbd28e9-ovn-node-metrics-cert\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030231 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.030726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030264 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-run\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030277 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030330 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-slash\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030360 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6d0e693-ccf2-4168-a63c-2f1428af95fe-dbus\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030387 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysctl-d\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030417 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-ovn\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030430 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-var-lib-kubelet\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030457 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysconfig\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030486 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-host\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030485 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-run\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030513 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-tmp\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030546 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030642 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-system-cni-dir\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030667 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-os-release\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030691 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.031623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030715 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-systemd\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030755 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ch9\" (UniqueName: \"kubernetes.io/projected/9f112c12-949a-414b-a960-8babbdbd28e9-kube-api-access-q4ch9\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030781 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-sys-fs\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030784 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030807 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkh2v\" (UniqueName: \"kubernetes.io/projected/120f7d41-d050-48d7-acd8-c9779c7af52f-kube-api-access-wkh2v\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030847 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-kubelet\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030851 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030901 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69cf0189-a6a2-406d-991b-35b69c360c09-konnectivity-ca\") pod \"konnectivity-agent-skhtl\" (UID: \"69cf0189-a6a2-406d-991b-35b69c360c09\") " pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030920 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6d0e693-ccf2-4168-a63c-2f1428af95fe-dbus\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030930 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6d0e693-ccf2-4168-a63c-2f1428af95fe-kubelet-config\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030957 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030973 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysconfig\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030984 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-registration-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030979 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8341f0c8-f991-4c19-a797-f7fce085608f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031021 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/120f7d41-d050-48d7-acd8-c9779c7af52f-host\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-host\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.032580 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/120f7d41-d050-48d7-acd8-c9779c7af52f-serviceca\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031063 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-var-lib-kubelet\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031053 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-sysctl-d\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031100 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031127 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-node-log\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031153 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-env-overrides\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031167 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-kubelet\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031201 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-var-lib-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031222 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031232 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-log-socket\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031260 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-cni-netd\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-cni-netd\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031318 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031356 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-ovnkube-config\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-ovnkube-script-lib\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.031400 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031417 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-systemd\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.033187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031442 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be264cf7-2ed3-4c99-8a85-264cacb645c7-hosts-file\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.031479 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs podName:69cffac7-1815-42f4-8bc7-ed8a8996b20f nodeName:}" failed. No retries permitted until 2026-07-29 15:46:31.531459888 +0000 UTC m=+2.072361366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs") pod "network-metrics-daemon-29fg9" (UID: "69cffac7-1815-42f4-8bc7-ed8a8996b20f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031544 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be264cf7-2ed3-4c99-8a85-264cacb645c7-hosts-file\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031587 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-system-cni-dir\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031667 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-os-release\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031712 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-run-systemd\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031728 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-node-log\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031829 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69cf0189-a6a2-406d-991b-35b69c360c09-konnectivity-ca\") pod \"konnectivity-agent-skhtl\" (UID: \"69cf0189-a6a2-406d-991b-35b69c360c09\") " pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031904 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6d0e693-ccf2-4168-a63c-2f1428af95fe-kubelet-config\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.030919 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-host-slash\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.031963 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-sys-fs\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.031995 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.032055 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret podName:c6d0e693-ccf2-4168-a63c-2f1428af95fe nodeName:}" failed. No retries permitted until 2026-07-29 15:46:31.532037958 +0000 UTC m=+2.072939410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret") pod "global-pull-secret-syncer-ptml9" (UID: "c6d0e693-ccf2-4168-a63c-2f1428af95fe") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032092 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/120f7d41-d050-48d7-acd8-c9779c7af52f-host\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032152 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7231bed-9e54-4bc2-8b99-c3053f2839b8-registration-dir\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032220 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-var-lib-openvswitch\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.033719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032400 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-env-overrides\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032486 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-systemd\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032538 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/120f7d41-d050-48d7-acd8-c9779c7af52f-serviceca\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032674 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f112c12-949a-414b-a960-8babbdbd28e9-log-socket\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032781 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8341f0c8-f991-4c19-a797-f7fce085608f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032806 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-etc-tuned\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032853 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69cf0189-a6a2-406d-991b-35b69c360c09-agent-certs\") pod \"konnectivity-agent-skhtl\" (UID: \"69cf0189-a6a2-406d-991b-35b69c360c09\") " pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.032933 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f112c12-949a-414b-a960-8babbdbd28e9-ovn-node-metrics-cert\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.033070 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-ovnkube-config\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.033103 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f112c12-949a-414b-a960-8babbdbd28e9-ovnkube-script-lib\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.034239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.033280 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-tmp\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.051401 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.051365 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkh2v\" (UniqueName: \"kubernetes.io/projected/120f7d41-d050-48d7-acd8-c9779c7af52f-kube-api-access-wkh2v\") pod \"node-ca-6mvd9\" (UID: \"120f7d41-d050-48d7-acd8-c9779c7af52f\") " pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:31.051625 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.051595 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrm2\" (UniqueName: \"kubernetes.io/projected/69cffac7-1815-42f4-8bc7-ed8a8996b20f-kube-api-access-6zrm2\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:31.051781 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.051750 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwzh\" (UniqueName: \"kubernetes.io/projected/be264cf7-2ed3-4c99-8a85-264cacb645c7-kube-api-access-lwwzh\") pod \"node-resolver-m7jv2\" (UID: \"be264cf7-2ed3-4c99-8a85-264cacb645c7\") " pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:31.052052 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.052033 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46p4\" (UniqueName: \"kubernetes.io/projected/fcdc6d9d-49c3-43af-b6bf-641dd6a05030-kube-api-access-s46p4\") pod \"tuned-6xb6w\" (UID: \"fcdc6d9d-49c3-43af-b6bf-641dd6a05030\") " pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.052202 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.052186 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ch9\" (UniqueName: \"kubernetes.io/projected/9f112c12-949a-414b-a960-8babbdbd28e9-kube-api-access-q4ch9\") pod \"ovnkube-node-nqdlk\" (UID: \"9f112c12-949a-414b-a960-8babbdbd28e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.052271 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.052248 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khh5\" (UniqueName: \"kubernetes.io/projected/c7231bed-9e54-4bc2-8b99-c3053f2839b8-kube-api-access-6khh5\") pod \"aws-ebs-csi-driver-node-tb5sv\" (UID: \"c7231bed-9e54-4bc2-8b99-c3053f2839b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.052786 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.052769 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xcdm\" (UniqueName: \"kubernetes.io/projected/8341f0c8-f991-4c19-a797-f7fce085608f-kube-api-access-4xcdm\") pod \"multus-additional-cni-plugins-8pzdz\" (UID: \"8341f0c8-f991-4c19-a797-f7fce085608f\") " pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.139658 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.139542 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bp8mc" Jul 29 15:46:31.148876 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.148848 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063034e5_79e7_4ffe_8748_75e0452a44e1.slice/crio-706d789a7ebf7b3da621f80c4d5ab801e3d5460efb786cffcf0e23338cc11aee WatchSource:0}: Error finding container 706d789a7ebf7b3da621f80c4d5ab801e3d5460efb786cffcf0e23338cc11aee: Status 404 returned error can't find the container with id 706d789a7ebf7b3da621f80c4d5ab801e3d5460efb786cffcf0e23338cc11aee Jul 29 15:46:31.155086 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.155065 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqlfg" Jul 29 15:46:31.163251 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.163223 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224b987f_fb05_4ebf_ac43_938f2f3b9340.slice/crio-cb79369db4ad138f8e2c11b597465bd8e253210cae6f7e4f26c7c92e4d5c7d0d WatchSource:0}: Error finding container cb79369db4ad138f8e2c11b597465bd8e253210cae6f7e4f26c7c92e4d5c7d0d: Status 404 returned error can't find the container with id cb79369db4ad138f8e2c11b597465bd8e253210cae6f7e4f26c7c92e4d5c7d0d Jul 29 15:46:31.170840 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.170818 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:31.178701 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.178675 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f112c12_949a_414b_a960_8babbdbd28e9.slice/crio-1557382a3bcbeea2b72eb84347ec2d40305906a9e38d8c8f4ef86fc4e99b55f3 WatchSource:0}: Error finding container 1557382a3bcbeea2b72eb84347ec2d40305906a9e38d8c8f4ef86fc4e99b55f3: Status 404 returned error can't find the container with id 1557382a3bcbeea2b72eb84347ec2d40305906a9e38d8c8f4ef86fc4e99b55f3 Jul 29 15:46:31.183289 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.183271 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:31.191469 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.191435 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cf0189_a6a2_406d_991b_35b69c360c09.slice/crio-14bb119edf36c3d9bea6ee4aa38ca4e8c12ec77928714651d060538b16984717 WatchSource:0}: Error finding container 14bb119edf36c3d9bea6ee4aa38ca4e8c12ec77928714651d060538b16984717: Status 404 returned error can't find the container with id 14bb119edf36c3d9bea6ee4aa38ca4e8c12ec77928714651d060538b16984717 Jul 29 15:46:31.208899 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.208851 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" Jul 29 15:46:31.217490 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.217456 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7231bed_9e54_4bc2_8b99_c3053f2839b8.slice/crio-d8e479afe3b3c647a9265a4a72fdbb0e88979061ac3cdbfcc708ad7fe694c2ae WatchSource:0}: Error finding container d8e479afe3b3c647a9265a4a72fdbb0e88979061ac3cdbfcc708ad7fe694c2ae: Status 404 returned error can't find the container with id d8e479afe3b3c647a9265a4a72fdbb0e88979061ac3cdbfcc708ad7fe694c2ae Jul 29 15:46:31.218454 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.218434 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" Jul 29 15:46:31.226148 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.226122 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdc6d9d_49c3_43af_b6bf_641dd6a05030.slice/crio-f03fe5d55b90d2241e72feac00ead89792b073a9fcc8ae248c4424ba1006d01f WatchSource:0}: Error finding container f03fe5d55b90d2241e72feac00ead89792b073a9fcc8ae248c4424ba1006d01f: Status 404 returned error can't find the container with id f03fe5d55b90d2241e72feac00ead89792b073a9fcc8ae248c4424ba1006d01f Jul 29 15:46:31.226771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.226751 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7jv2" Jul 29 15:46:31.236247 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.236222 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6mvd9" Jul 29 15:46:31.236900 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.236877 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe264cf7_2ed3_4c99_8a85_264cacb645c7.slice/crio-2ad2ac334ce3712bc28fb592500be4af76b56b57a4aa7e06fc79ed8add1faa14 WatchSource:0}: Error finding container 2ad2ac334ce3712bc28fb592500be4af76b56b57a4aa7e06fc79ed8add1faa14: Status 404 returned error can't find the container with id 2ad2ac334ce3712bc28fb592500be4af76b56b57a4aa7e06fc79ed8add1faa14 Jul 29 15:46:31.241781 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.241760 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" Jul 29 15:46:31.245037 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.245015 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120f7d41_d050_48d7_acd8_c9779c7af52f.slice/crio-6e3c28733e6fbd94ab0f1c8048c3076b8d93726006e43b1aec061fe4b93cc8d5 WatchSource:0}: Error finding container 6e3c28733e6fbd94ab0f1c8048c3076b8d93726006e43b1aec061fe4b93cc8d5: Status 404 returned error can't find the container with id 6e3c28733e6fbd94ab0f1c8048c3076b8d93726006e43b1aec061fe4b93cc8d5 Jul 29 15:46:31.250233 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:46:31.250208 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8341f0c8_f991_4c19_a797_f7fce085608f.slice/crio-afafc81c44d3e52a0eb3a2d1b6acc8b41695beaa76efdabdf3f3bd509bc624a7 WatchSource:0}: Error finding container afafc81c44d3e52a0eb3a2d1b6acc8b41695beaa76efdabdf3f3bd509bc624a7: Status 404 returned error can't find the container with id afafc81c44d3e52a0eb3a2d1b6acc8b41695beaa76efdabdf3f3bd509bc624a7 Jul 29 15:46:31.535019 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.534924 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:31.535019 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.534967 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.535029 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535127 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535160 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535165 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535182 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535197 2555 projected.go:196] Error preparing data for projected volume kube-api-access-2xjwl for pod openshift-network-diagnostics/network-check-target-gqd46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535221 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret podName:c6d0e693-ccf2-4168-a63c-2f1428af95fe nodeName:}" failed. No retries permitted until 2026-07-29 15:46:32.535196354 +0000 UTC m=+3.076097852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret") pod "global-pull-secret-syncer-ptml9" (UID: "c6d0e693-ccf2-4168-a63c-2f1428af95fe") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535246 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs podName:69cffac7-1815-42f4-8bc7-ed8a8996b20f nodeName:}" failed. No retries permitted until 2026-07-29 15:46:32.535234409 +0000 UTC m=+3.076135863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs") pod "network-metrics-daemon-29fg9" (UID: "69cffac7-1815-42f4-8bc7-ed8a8996b20f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:31.535324 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.535263 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl podName:02e74657-8c54-477b-a184-eb2a35c8eff5 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:32.5352543 +0000 UTC m=+3.076155755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xjwl" (UniqueName: "kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl") pod "network-check-target-gqd46" (UID: "02e74657-8c54-477b-a184-eb2a35c8eff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:31.660024 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.658812 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:31.877811 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.877625 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 15:41:31 +0000 UTC" deadline="2028-03-14 12:55:26.007301819 +0000 UTC" Jul 29 15:46:31.877811 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.877689 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14253h8m54.129616866s" Jul 29 15:46:31.918386 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.918342 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:31.918552 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:31.918491 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:31.929053 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.928962 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerStarted","Data":"afafc81c44d3e52a0eb3a2d1b6acc8b41695beaa76efdabdf3f3bd509bc624a7"} Jul 29 15:46:31.934873 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.934774 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7jv2" event={"ID":"be264cf7-2ed3-4c99-8a85-264cacb645c7","Type":"ContainerStarted","Data":"2ad2ac334ce3712bc28fb592500be4af76b56b57a4aa7e06fc79ed8add1faa14"} Jul 29 15:46:31.939824 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.939704 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" event={"ID":"fcdc6d9d-49c3-43af-b6bf-641dd6a05030","Type":"ContainerStarted","Data":"f03fe5d55b90d2241e72feac00ead89792b073a9fcc8ae248c4424ba1006d01f"} Jul 29 15:46:31.942023 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.941929 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" event={"ID":"c7231bed-9e54-4bc2-8b99-c3053f2839b8","Type":"ContainerStarted","Data":"d8e479afe3b3c647a9265a4a72fdbb0e88979061ac3cdbfcc708ad7fe694c2ae"} Jul 29 15:46:31.944426 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.944309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"1557382a3bcbeea2b72eb84347ec2d40305906a9e38d8c8f4ef86fc4e99b55f3"} Jul 29 15:46:31.946994 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.946910 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqlfg" event={"ID":"224b987f-fb05-4ebf-ac43-938f2f3b9340","Type":"ContainerStarted","Data":"cb79369db4ad138f8e2c11b597465bd8e253210cae6f7e4f26c7c92e4d5c7d0d"} Jul 29 15:46:31.949295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.949209 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6mvd9" event={"ID":"120f7d41-d050-48d7-acd8-c9779c7af52f","Type":"ContainerStarted","Data":"6e3c28733e6fbd94ab0f1c8048c3076b8d93726006e43b1aec061fe4b93cc8d5"} Jul 29 15:46:31.952173 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.952094 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-skhtl" event={"ID":"69cf0189-a6a2-406d-991b-35b69c360c09","Type":"ContainerStarted","Data":"14bb119edf36c3d9bea6ee4aa38ca4e8c12ec77928714651d060538b16984717"} Jul 29 15:46:31.954232 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:31.954207 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bp8mc" event={"ID":"063034e5-79e7-4ffe-8748-75e0452a44e1","Type":"ContainerStarted","Data":"706d789a7ebf7b3da621f80c4d5ab801e3d5460efb786cffcf0e23338cc11aee"} Jul 29 15:46:32.542864 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.542821 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:32.543068 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.542879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:32.543068 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.542944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:32.543195 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543097 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:32.543195 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543120 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:32.543195 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543134 2555 projected.go:196] Error preparing data for projected volume kube-api-access-2xjwl for pod openshift-network-diagnostics/network-check-target-gqd46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:32.543340 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543201 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl podName:02e74657-8c54-477b-a184-eb2a35c8eff5 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:34.543180285 +0000 UTC m=+5.084081738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xjwl" (UniqueName: "kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl") pod "network-check-target-gqd46" (UID: "02e74657-8c54-477b-a184-eb2a35c8eff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:32.543719 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543695 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:32.543848 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543758 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret podName:c6d0e693-ccf2-4168-a63c-2f1428af95fe nodeName:}" failed. No retries permitted until 2026-07-29 15:46:34.54374123 +0000 UTC m=+5.084642699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret") pod "global-pull-secret-syncer-ptml9" (UID: "c6d0e693-ccf2-4168-a63c-2f1428af95fe") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:32.543848 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543818 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:32.543966 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.543854 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs podName:69cffac7-1815-42f4-8bc7-ed8a8996b20f nodeName:}" failed. No retries permitted until 2026-07-29 15:46:34.543842807 +0000 UTC m=+5.084744255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs") pod "network-metrics-daemon-29fg9" (UID: "69cffac7-1815-42f4-8bc7-ed8a8996b20f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:32.771691 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.771649 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:32.876581 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.876492 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:32.878018 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.877982 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 15:41:31 +0000 UTC" deadline="2028-01-02 14:06:47.191548963 +0000 UTC" Jul 29 15:46:32.878018 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.878017 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12526h20m14.313536549s" Jul 29 15:46:32.918579 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.918543 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:32.918788 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.918715 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:32.919189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:32.919167 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:32.919298 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:32.919273 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:33.920747 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:33.920643 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:33.921232 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:33.920802 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:34.565679 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:34.565389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:34.565870 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:34.565710 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:34.565870 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:34.565747 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:34.566098 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.565925 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:34.566098 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.566010 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs podName:69cffac7-1815-42f4-8bc7-ed8a8996b20f nodeName:}" failed. No retries permitted until 2026-07-29 15:46:38.565987116 +0000 UTC m=+9.106888584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs") pod "network-metrics-daemon-29fg9" (UID: "69cffac7-1815-42f4-8bc7-ed8a8996b20f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:34.566491 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.566471 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:34.566491 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.566496 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:34.566672 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.566510 2555 projected.go:196] Error preparing data for projected volume kube-api-access-2xjwl for pod openshift-network-diagnostics/network-check-target-gqd46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:34.566672 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.566568 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl podName:02e74657-8c54-477b-a184-eb2a35c8eff5 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:38.566550502 +0000 UTC m=+9.107451965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xjwl" (UniqueName: "kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl") pod "network-check-target-gqd46" (UID: "02e74657-8c54-477b-a184-eb2a35c8eff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:34.566672 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.566635 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:34.566839 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.566688 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret podName:c6d0e693-ccf2-4168-a63c-2f1428af95fe nodeName:}" failed. No retries permitted until 2026-07-29 15:46:38.566675071 +0000 UTC m=+9.107576521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret") pod "global-pull-secret-syncer-ptml9" (UID: "c6d0e693-ccf2-4168-a63c-2f1428af95fe") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:34.918304 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:34.918224 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:34.918304 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:34.918271 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:34.918531 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.918365 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:34.918874 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:34.918844 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:34.972398 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:34.972353 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" event={"ID":"f09a92c41c6e0b2866952bb93aa6998f","Type":"ContainerStarted","Data":"31e219a098c488f3ab7b5fb674aa270d32749a5fa32a0c82833f81677d328037"} Jul 29 15:46:35.922734 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:35.922220 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:35.922734 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:35.922339 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:36.918485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:36.918447 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:36.918976 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:36.918496 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:36.918976 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:36.918604 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:36.918976 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:36.918714 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:37.919103 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:37.919067 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:37.919600 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:37.919207 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:38.602691 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:38.602749 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:38.602779 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.602896 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.602901 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.602934 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.602947 2555 projected.go:196] Error preparing data for projected volume kube-api-access-2xjwl for pod openshift-network-diagnostics/network-check-target-gqd46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.602954 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.602963 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs podName:69cffac7-1815-42f4-8bc7-ed8a8996b20f nodeName:}" failed. No retries permitted until 2026-07-29 15:46:46.602942009 +0000 UTC m=+17.143843460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs") pod "network-metrics-daemon-29fg9" (UID: "69cffac7-1815-42f4-8bc7-ed8a8996b20f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.603020 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl podName:02e74657-8c54-477b-a184-eb2a35c8eff5 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:46.603000642 +0000 UTC m=+17.143902102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xjwl" (UniqueName: "kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl") pod "network-check-target-gqd46" (UID: "02e74657-8c54-477b-a184-eb2a35c8eff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:38.603317 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.603038 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret podName:c6d0e693-ccf2-4168-a63c-2f1428af95fe nodeName:}" failed. No retries permitted until 2026-07-29 15:46:46.603027845 +0000 UTC m=+17.143929307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret") pod "global-pull-secret-syncer-ptml9" (UID: "c6d0e693-ccf2-4168-a63c-2f1428af95fe") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:38.918925 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:38.918843 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:38.919100 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.919044 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:38.919100 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:38.919064 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:38.919534 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:38.919204 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:39.919162 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:39.919128 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:39.919628 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:39.919240 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:40.918930 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:40.918892 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:40.919104 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:40.918892 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:40.919104 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:40.919039 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:40.919226 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:40.919143 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:40.986991 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:40.986939 2555 generic.go:356] "Generic (PLEG): container finished" podID="f09a92c41c6e0b2866952bb93aa6998f" containerID="31e219a098c488f3ab7b5fb674aa270d32749a5fa32a0c82833f81677d328037" exitCode=0 Jul 29 15:46:40.987170 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:40.986994 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" event={"ID":"f09a92c41c6e0b2866952bb93aa6998f","Type":"ContainerDied","Data":"31e219a098c488f3ab7b5fb674aa270d32749a5fa32a0c82833f81677d328037"} Jul 29 15:46:41.918732 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:41.918696 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:41.918926 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:41.918842 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:42.918170 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:42.918140 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:42.918557 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:42.918145 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:42.918557 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:42.918234 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:42.918557 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:42.918316 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:43.918246 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:43.918156 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:43.918723 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:43.918296 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:44.918666 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:44.918633 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:44.918666 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:44.918670 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:44.919193 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:44.918765 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:44.919193 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:44.918905 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:45.918765 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:45.918730 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:45.919160 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:45.918842 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:46.665161 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:46.665122 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:46.665161 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:46.665180 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665279 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:46.665302 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665309 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665351 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret podName:c6d0e693-ccf2-4168-a63c-2f1428af95fe nodeName:}" failed. No retries permitted until 2026-07-29 15:47:02.66533262 +0000 UTC m=+33.206234067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret") pod "global-pull-secret-syncer-ptml9" (UID: "c6d0e693-ccf2-4168-a63c-2f1428af95fe") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665357 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665375 2555 projected.go:196] Error preparing data for projected volume kube-api-access-2xjwl for pod openshift-network-diagnostics/network-check-target-gqd46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665358 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665416 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl podName:02e74657-8c54-477b-a184-eb2a35c8eff5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:02.665405784 +0000 UTC m=+33.206307249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xjwl" (UniqueName: "kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl") pod "network-check-target-gqd46" (UID: "02e74657-8c54-477b-a184-eb2a35c8eff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:46.665475 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.665433 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs podName:69cffac7-1815-42f4-8bc7-ed8a8996b20f nodeName:}" failed. No retries permitted until 2026-07-29 15:47:02.665422314 +0000 UTC m=+33.206323762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs") pod "network-metrics-daemon-29fg9" (UID: "69cffac7-1815-42f4-8bc7-ed8a8996b20f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:46.918779 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:46.918689 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:46.919220 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.918805 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:46.919220 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:46.918868 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:46.919220 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:46.918987 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:47.918234 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:47.918198 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:47.918449 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:47.918327 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:48.918923 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:48.918886 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:48.919391 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:48.918886 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:48.919391 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:48.918999 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:48.919391 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:48.919138 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:49.919101 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:49.919061 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:49.919469 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:49.919185 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:50.918689 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:50.918649 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:50.918689 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:50.918687 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:50.918919 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:50.918788 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:50.918975 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:50.918913 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:51.918797 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:51.918422 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:51.919448 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:51.918883 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:52.009568 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.009490 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7jv2" event={"ID":"be264cf7-2ed3-4c99-8a85-264cacb645c7","Type":"ContainerStarted","Data":"24b2b81623e230d1a9eac528b7f0cf2a5bd123f290186b2e37d299ef39a9dfc5"} Jul 29 15:46:52.010947 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.010920 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" event={"ID":"fcdc6d9d-49c3-43af-b6bf-641dd6a05030","Type":"ContainerStarted","Data":"37331186172c219e9578550c5936e703193aaec0982851289121f0387b2239b5"} Jul 29 15:46:52.012301 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.012269 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" event={"ID":"c7231bed-9e54-4bc2-8b99-c3053f2839b8","Type":"ContainerStarted","Data":"4924fb0d4b285d32408a73bd7a717bb15b3c24e27a183f818c97f93758610fae"} Jul 29 15:46:52.015489 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.015461 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"2a9f08ebeec690c743c9059d706a44621bc4009426628ccdda9b09fdd8fe4c0e"} Jul 29 15:46:52.015645 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.015526 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"cce5fc64779f43dfd2a7c10ba51bc272a7fe58c891cdb8c21a98cd63381caa01"} Jul 29 15:46:52.015645 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.015570 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"705bd84a3afa57128532d44e144d6d33cf620dfa7230b51fa29ae083e052d976"} Jul 29 15:46:52.015645 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.015634 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"c79aec00b2b9ebb9f295a637f503e6913f9e13bbca4d94825d4b9b12a674a871"} Jul 29 15:46:52.015796 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.015664 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"5f6494596913aa3d1ae6a98c496135ba4dba311d5f3449f33a28ee1bad5fe09b"} Jul 29 15:46:52.015796 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.015693 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"896f3fc21db07481dd064b78667fea7ce09246583061d5864355a4ce2ebc441c"} Jul 29 15:46:52.016962 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.016938 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" event={"ID":"a573c2ec0c2e9c2aaedfbe717cd64a75","Type":"ContainerStarted","Data":"80114d2eefb75d5b1a370f08d979d870e83e76c8fca40c291822c1ab70424b08"} Jul 29 15:46:52.018434 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.018415 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6mvd9" event={"ID":"120f7d41-d050-48d7-acd8-c9779c7af52f","Type":"ContainerStarted","Data":"ee0616c1b55d9a310d4d771dacc7806b534f25e0d6fd1156ca3f33aa47a9cad6"} Jul 29 15:46:52.019842 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.019823 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-skhtl" event={"ID":"69cf0189-a6a2-406d-991b-35b69c360c09","Type":"ContainerStarted","Data":"df2ebcbfc111bd740a39db38ba18d3504c798ea376024b4545aeeababe79e5ca"} Jul 29 15:46:52.021446 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.021421 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bp8mc" event={"ID":"063034e5-79e7-4ffe-8748-75e0452a44e1","Type":"ContainerStarted","Data":"79e62827965063e71e137ee3ccb141fc2a57aa11b6f5f4ad47504e5736411392"} Jul 29 15:46:52.023164 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.023146 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" event={"ID":"f09a92c41c6e0b2866952bb93aa6998f","Type":"ContainerStarted","Data":"2f1f0eb08e425f91e89e615407e25fb051db3cd04ac0c4c914d926e8321cc860"} Jul 29 15:46:52.024546 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.024525 2555 generic.go:356] "Generic (PLEG): container finished" podID="8341f0c8-f991-4c19-a797-f7fce085608f" containerID="52794719ce4b2461c3b3715a0df1db4eb45148825ef3aa15ce9cdc862a912cb7" exitCode=0 Jul 29 15:46:52.024643 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.024573 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerDied","Data":"52794719ce4b2461c3b3715a0df1db4eb45148825ef3aa15ce9cdc862a912cb7"} Jul 29 15:46:52.029879 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.029839 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m7jv2" podStartSLOduration=2.055574726 podStartE2EDuration="22.02982698s" totalImagesPullingTime="19.974252254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:52.029809787 +0000 UTC m=+22.570711256" watchObservedRunningTime="2026-07-29 15:46:52.02982698 +0000 UTC m=+22.570728445" Jul 29 15:46:52.047571 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.047512 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6mvd9" podStartSLOduration=2.094478398 podStartE2EDuration="22.047497705s" totalImagesPullingTime="19.953019307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:52.047390948 +0000 UTC m=+22.588292418" watchObservedRunningTime="2026-07-29 15:46:52.047497705 +0000 UTC m=+22.588399174" Jul 29 15:46:52.072341 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.072296 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" podStartSLOduration=21.072279794 podStartE2EDuration="21.072279794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:52.071984876 +0000 UTC m=+22.612886346" watchObservedRunningTime="2026-07-29 15:46:52.072279794 +0000 UTC m=+22.613181262" Jul 29 15:46:52.090939 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.090898 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-skhtl" podStartSLOduration=6.116071528 podStartE2EDuration="22.090881079s" totalImagesPullingTime="15.974809551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:52.090769826 +0000 UTC m=+22.631671294" watchObservedRunningTime="2026-07-29 15:46:52.090881079 +0000 UTC m=+22.631782575" Jul 29 15:46:52.120481 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.120433 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6xb6w" podStartSLOduration=2.142825747 podStartE2EDuration="22.120418007s" totalImagesPullingTime="19.97759226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:52.120291487 +0000 UTC m=+22.661192965" watchObservedRunningTime="2026-07-29 15:46:52.120418007 +0000 UTC m=+22.661319476" Jul 29 15:46:52.227461 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.227419 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" podStartSLOduration=21.227404063 podStartE2EDuration="21.227404063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:52.197460069 +0000 UTC m=+22.738361537" watchObservedRunningTime="2026-07-29 15:46:52.227404063 +0000 UTC m=+22.768305528" Jul 29 15:46:52.227836 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.227814 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bp8mc" podStartSLOduration=2.114165314 podStartE2EDuration="22.227807953s" totalImagesPullingTime="20.113642639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:52.227276882 +0000 UTC m=+22.768178351" watchObservedRunningTime="2026-07-29 15:46:52.227807953 +0000 UTC m=+22.768709422" Jul 29 15:46:52.918694 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.918656 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:52.918911 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:52.918664 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:52.918911 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:52.918791 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:52.918911 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:52.918851 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:53.028838 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:53.028794 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqlfg" event={"ID":"224b987f-fb05-4ebf-ac43-938f2f3b9340","Type":"ContainerStarted","Data":"e6dd342a2b6cf064a1e94916175f5f14e3005a40ba4cc16f673aa706641ac463"} Jul 29 15:46:53.131693 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:53.131569 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 15:46:53.897615 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:53.897253 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T15:46:53.131612616Z","UUID":"0337d566-1123-4079-b726-ba4eac2f3be6","Handler":null,"Name":"","Endpoint":""} Jul 29 15:46:53.899038 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:53.898997 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 15:46:53.899038 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:53.899033 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 15:46:53.918882 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:53.918847 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:53.919057 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:53.918980 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:54.032660 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:54.032624 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" event={"ID":"c7231bed-9e54-4bc2-8b99-c3053f2839b8","Type":"ContainerStarted","Data":"de3b4c5f7dcd88fb076526102de821b572f5234b898b8a208688f32fa3e51af0"} Jul 29 15:46:54.855100 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:54.855061 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:54.855784 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:54.855755 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:54.873058 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:54.872926 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vqlfg" podStartSLOduration=4.835446176 podStartE2EDuration="24.872904869s" totalImagesPullingTime="20.037458693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:53.046356434 +0000 UTC m=+23.587257918" watchObservedRunningTime="2026-07-29 15:46:54.872904869 +0000 UTC m=+25.413806340" Jul 29 15:46:54.918064 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:54.918032 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:54.918247 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:54.918127 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:54.918247 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:54.918189 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:54.918359 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:54.918301 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:55.037280 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:55.037244 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" event={"ID":"c7231bed-9e54-4bc2-8b99-c3053f2839b8","Type":"ContainerStarted","Data":"77cb5c8b71b2fce3024d78721071909e40c4e0ce821669638116b6ed7df87288"} Jul 29 15:46:55.041649 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:55.041607 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"916d032ef115e0e22fe5439dbfd5d5e795b95992bdfd13c547d6c0c9bad1dd75"} Jul 29 15:46:55.041807 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:55.041725 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:55.042410 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:55.042389 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-skhtl" Jul 29 15:46:55.058916 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:55.058858 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tb5sv" podStartSLOduration=2.504153854 podStartE2EDuration="25.058839106s" totalImagesPullingTime="22.554685252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:55.057986307 +0000 UTC m=+25.598887777" watchObservedRunningTime="2026-07-29 15:46:55.058839106 +0000 UTC m=+25.599740575" Jul 29 15:46:55.918087 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:55.918041 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:55.918256 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:55.918160 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:56.918598 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:56.918394 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:56.919333 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:56.918394 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:56.919333 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:56.918686 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:56.919333 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:56.918729 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:57.047533 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.047501 2555 generic.go:356] "Generic (PLEG): container finished" podID="8341f0c8-f991-4c19-a797-f7fce085608f" containerID="38751910fb2b6c4ea47eee36e803bfd584143922df131d4ac2c31fdfd2245977" exitCode=0 Jul 29 15:46:57.047727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.047567 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerDied","Data":"38751910fb2b6c4ea47eee36e803bfd584143922df131d4ac2c31fdfd2245977"} Jul 29 15:46:57.051414 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.051394 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" event={"ID":"9f112c12-949a-414b-a960-8babbdbd28e9","Type":"ContainerStarted","Data":"18277262645a7d4ba2d95026db1dc79a90b1427657c2fc9ab7b39e4be71a47a5"} Jul 29 15:46:57.051827 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.051810 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:57.051920 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.051881 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:57.066857 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.066824 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:57.101858 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.101803 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" podStartSLOduration=7.013861199 podStartE2EDuration="27.101786512s" totalImagesPullingTime="20.087925313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:46:57.101446009 +0000 UTC m=+27.642347478" watchObservedRunningTime="2026-07-29 15:46:57.101786512 +0000 UTC m=+27.642687980" Jul 29 15:46:57.918319 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:57.918281 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:57.918551 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:57.918424 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:58.054498 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.054302 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:58.070792 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.070754 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:46:58.515649 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.515611 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gqd46"] Jul 29 15:46:58.515977 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.515743 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:58.515977 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:58.515847 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:58.517775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.517749 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ptml9"] Jul 29 15:46:58.517903 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.517848 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:58.517976 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:58.517956 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:46:58.523708 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.523683 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29fg9"] Jul 29 15:46:58.523822 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:58.523785 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:46:58.523884 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:58.523863 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:46:59.058568 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:59.058529 2555 generic.go:356] "Generic (PLEG): container finished" podID="8341f0c8-f991-4c19-a797-f7fce085608f" containerID="fad61c1103aff23f0aa97331f8df2e29f062f3b86f36260aa73269a32862d622" exitCode=0 Jul 29 15:46:59.058963 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:59.058650 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerDied","Data":"fad61c1103aff23f0aa97331f8df2e29f062f3b86f36260aa73269a32862d622"} Jul 29 15:46:59.918882 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:59.918846 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:46:59.919071 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:59.918947 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:46:59.919071 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:46:59.918990 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:46:59.919377 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:46:59.919088 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:47:00.918310 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:00.918219 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:47:00.918764 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:00.918329 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:47:01.066321 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:01.066289 2555 generic.go:356] "Generic (PLEG): container finished" podID="8341f0c8-f991-4c19-a797-f7fce085608f" containerID="ae810b03f6b4382d4123a37bc9ecfcc4da639285b731283adc4c7d413e62f28e" exitCode=0 Jul 29 15:47:01.066481 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:01.066348 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerDied","Data":"ae810b03f6b4382d4123a37bc9ecfcc4da639285b731283adc4c7d413e62f28e"} Jul 29 15:47:01.918520 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:01.918475 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:01.919003 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:01.918679 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:47:01.919003 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:01.918728 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:47:01.919003 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:01.918840 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:47:02.688946 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:02.688903 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:47:02.688946 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:02.688952 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:02.689000 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689081 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689132 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689152 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret podName:c6d0e693-ccf2-4168-a63c-2f1428af95fe nodeName:}" failed. No retries permitted until 2026-07-29 15:47:34.689136415 +0000 UTC m=+65.230037862 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret") pod "global-pull-secret-syncer-ptml9" (UID: "c6d0e693-ccf2-4168-a63c-2f1428af95fe") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689152 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689164 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689177 2555 projected.go:196] Error preparing data for projected volume kube-api-access-2xjwl for pod openshift-network-diagnostics/network-check-target-gqd46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:47:02.689198 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689203 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl podName:02e74657-8c54-477b-a184-eb2a35c8eff5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:34.689197158 +0000 UTC m=+65.230098605 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xjwl" (UniqueName: "kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl") pod "network-check-target-gqd46" (UID: "02e74657-8c54-477b-a184-eb2a35c8eff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:47:02.689619 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.689233 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs podName:69cffac7-1815-42f4-8bc7-ed8a8996b20f nodeName:}" failed. No retries permitted until 2026-07-29 15:47:34.689214856 +0000 UTC m=+65.230116319 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs") pod "network-metrics-daemon-29fg9" (UID: "69cffac7-1815-42f4-8bc7-ed8a8996b20f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:47:02.918283 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:02.918199 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:47:02.918487 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:02.918357 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29fg9" podUID="69cffac7-1815-42f4-8bc7-ed8a8996b20f" Jul 29 15:47:03.918490 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:03.918452 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:03.919006 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:03.918598 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqd46" podUID="02e74657-8c54-477b-a184-eb2a35c8eff5" Jul 29 15:47:03.919006 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:03.918613 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:47:03.919006 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:03.918700 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptml9" podUID="c6d0e693-ccf2-4168-a63c-2f1428af95fe" Jul 29 15:47:04.264872 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.264661 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeReady" Jul 29 15:47:04.265058 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.265008 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 15:47:04.302146 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.302113 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-768645b995-46b4w"] Jul 29 15:47:04.305578 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.305556 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.308319 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.308290 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 15:47:04.308446 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.308392 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgnpm\"" Jul 29 15:47:04.308794 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.308576 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 15:47:04.308794 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.308577 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 15:47:04.313826 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.313802 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 15:47:04.316992 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.316968 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xmzml"] Jul 29 15:47:04.319020 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.318997 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.321145 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.321110 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-768645b995-46b4w"] Jul 29 15:47:04.321469 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.321440 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 15:47:04.321566 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.321535 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 15:47:04.321722 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.321571 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx57w\"" Jul 29 15:47:04.321888 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.321862 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l7g6v"] Jul 29 15:47:04.323829 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.323809 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.325943 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.325921 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 15:47:04.326479 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.326426 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h46bm\"" Jul 29 15:47:04.326479 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.326454 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 15:47:04.326660 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.326540 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 15:47:04.332669 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.332632 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmzml"] Jul 29 15:47:04.336374 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.335892 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l7g6v"] Jul 29 15:47:04.402318 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402275 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402496 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402327 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-installation-pull-secrets\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402496 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402352 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-bound-sa-token\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402496 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402425 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571cee9d-30cc-4291-91c0-abb9ef90f3c5-tmp-dir\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.402496 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/846b9e1c-7ded-4a38-86ad-61f753398cf6-cert\") pod \"ingress-canary-l7g6v\" (UID: \"846b9e1c-7ded-4a38-86ad-61f753398cf6\") " pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.402716 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402506 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-image-registry-private-configuration\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402716 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96grv\" (UniqueName: \"kubernetes.io/projected/571cee9d-30cc-4291-91c0-abb9ef90f3c5-kube-api-access-96grv\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.402716 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402600 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck87x\" (UniqueName: \"kubernetes.io/projected/846b9e1c-7ded-4a38-86ad-61f753398cf6-kube-api-access-ck87x\") pod \"ingress-canary-l7g6v\" (UID: \"846b9e1c-7ded-4a38-86ad-61f753398cf6\") " pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.402716 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402699 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4de6cfda-e440-4046-87eb-2f0ca0c54470-ca-trust-extracted\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402738 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-trusted-ca\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402774 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-certificates\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402837 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8sn7\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-kube-api-access-q8sn7\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.402958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402861 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.402958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.402889 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571cee9d-30cc-4291-91c0-abb9ef90f3c5-config-volume\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.503713 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.503657 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.503897 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.503719 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-installation-pull-secrets\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.503897 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.503746 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-bound-sa-token\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.503897 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:04.503843 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 15:47:04.503897 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:04.503863 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768645b995-46b4w: secret "image-registry-tls" not found Jul 29 15:47:04.504057 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:04.503927 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls podName:4de6cfda-e440-4046-87eb-2f0ca0c54470 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:05.003906603 +0000 UTC m=+35.544808074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls") pod "image-registry-768645b995-46b4w" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470") : secret "image-registry-tls" not found Jul 29 15:47:04.504057 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.503946 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571cee9d-30cc-4291-91c0-abb9ef90f3c5-tmp-dir\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.504057 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.503976 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/846b9e1c-7ded-4a38-86ad-61f753398cf6-cert\") pod \"ingress-canary-l7g6v\" (UID: \"846b9e1c-7ded-4a38-86ad-61f753398cf6\") " pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.504057 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504006 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-image-registry-private-configuration\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.504057 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504031 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96grv\" (UniqueName: \"kubernetes.io/projected/571cee9d-30cc-4291-91c0-abb9ef90f3c5-kube-api-access-96grv\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.504284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504062 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck87x\" (UniqueName: \"kubernetes.io/projected/846b9e1c-7ded-4a38-86ad-61f753398cf6-kube-api-access-ck87x\") pod \"ingress-canary-l7g6v\" (UID: \"846b9e1c-7ded-4a38-86ad-61f753398cf6\") " pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.504284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504099 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4de6cfda-e440-4046-87eb-2f0ca0c54470-ca-trust-extracted\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.504284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504136 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-trusted-ca\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.504284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504170 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-certificates\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.504284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504214 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8sn7\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-kube-api-access-q8sn7\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.504284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504241 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.504284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504269 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571cee9d-30cc-4291-91c0-abb9ef90f3c5-config-volume\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.504830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504290 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571cee9d-30cc-4291-91c0-abb9ef90f3c5-tmp-dir\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.504830 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4de6cfda-e440-4046-87eb-2f0ca0c54470-ca-trust-extracted\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.504830 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:04.504710 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:04.504830 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:04.504774 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls podName:571cee9d-30cc-4291-91c0-abb9ef90f3c5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:05.00475363 +0000 UTC m=+35.545655094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls") pod "dns-default-xmzml" (UID: "571cee9d-30cc-4291-91c0-abb9ef90f3c5") : secret "dns-default-metrics-tls" not found Jul 29 15:47:04.505067 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504862 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571cee9d-30cc-4291-91c0-abb9ef90f3c5-config-volume\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.505067 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.504961 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-certificates\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.505369 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.505335 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-trusted-ca\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.508795 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.508771 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-installation-pull-secrets\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.508795 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.508787 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/846b9e1c-7ded-4a38-86ad-61f753398cf6-cert\") pod \"ingress-canary-l7g6v\" (UID: \"846b9e1c-7ded-4a38-86ad-61f753398cf6\") " pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.508982 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.508794 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-image-registry-private-configuration\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.517738 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.517669 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-bound-sa-token\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.517738 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.517720 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96grv\" (UniqueName: \"kubernetes.io/projected/571cee9d-30cc-4291-91c0-abb9ef90f3c5-kube-api-access-96grv\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:04.517946 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.517720 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck87x\" (UniqueName: \"kubernetes.io/projected/846b9e1c-7ded-4a38-86ad-61f753398cf6-kube-api-access-ck87x\") pod \"ingress-canary-l7g6v\" (UID: \"846b9e1c-7ded-4a38-86ad-61f753398cf6\") " pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.518632 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.518609 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8sn7\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-kube-api-access-q8sn7\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:04.640405 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.640363 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l7g6v" Jul 29 15:47:04.918999 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.918919 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:47:04.921844 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.921815 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 15:47:04.921975 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:04.921866 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28zhr\"" Jul 29 15:47:05.007787 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.007736 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:05.007972 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.007830 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:05.007972 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:05.007919 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 15:47:05.007972 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:05.007937 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768645b995-46b4w: secret "image-registry-tls" not found Jul 29 15:47:05.008129 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:05.007986 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls podName:4de6cfda-e440-4046-87eb-2f0ca0c54470 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:06.00796941 +0000 UTC m=+36.548870857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls") pod "image-registry-768645b995-46b4w" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470") : secret "image-registry-tls" not found Jul 29 15:47:05.008129 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:05.007924 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:05.008129 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:05.008076 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls podName:571cee9d-30cc-4291-91c0-abb9ef90f3c5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:06.008056595 +0000 UTC m=+36.548958272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls") pod "dns-default-xmzml" (UID: "571cee9d-30cc-4291-91c0-abb9ef90f3c5") : secret "dns-default-metrics-tls" not found Jul 29 15:47:05.303849 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.303818 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l7g6v"] Jul 29 15:47:05.307178 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:05.307146 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846b9e1c_7ded_4a38_86ad_61f753398cf6.slice/crio-362f18451142ca88685d6d56e6b84ab618fad7966ccf279d6ff8e63250c8c46e WatchSource:0}: Error finding container 362f18451142ca88685d6d56e6b84ab618fad7966ccf279d6ff8e63250c8c46e: Status 404 returned error can't find the container with id 362f18451142ca88685d6d56e6b84ab618fad7966ccf279d6ff8e63250c8c46e Jul 29 15:47:05.918349 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.918315 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:47:05.918541 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.918503 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:05.921282 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.921247 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 15:47:05.922153 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.921295 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 15:47:05.922153 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.921567 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hf4f8\"" Jul 29 15:47:05.922417 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:05.922400 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 15:47:06.014968 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:06.014938 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:06.015165 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:06.015006 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:06.015165 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:06.015104 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 15:47:06.015165 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:06.015129 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768645b995-46b4w: secret "image-registry-tls" not found Jul 29 15:47:06.015323 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:06.015188 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls podName:4de6cfda-e440-4046-87eb-2f0ca0c54470 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:08.015171062 +0000 UTC m=+38.556072510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls") pod "image-registry-768645b995-46b4w" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470") : secret "image-registry-tls" not found Jul 29 15:47:06.015323 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:06.015112 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:06.015323 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:06.015214 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls podName:571cee9d-30cc-4291-91c0-abb9ef90f3c5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:08.015208079 +0000 UTC m=+38.556109527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls") pod "dns-default-xmzml" (UID: "571cee9d-30cc-4291-91c0-abb9ef90f3c5") : secret "dns-default-metrics-tls" not found Jul 29 15:47:06.083227 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:06.083188 2555 generic.go:356] "Generic (PLEG): container finished" podID="8341f0c8-f991-4c19-a797-f7fce085608f" containerID="73d2fc8d7901ebf166f9199b7f69b6f4b5ce49ce2e062b6b36eb895259e17403" exitCode=0 Jul 29 15:47:06.083412 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:06.083280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerDied","Data":"73d2fc8d7901ebf166f9199b7f69b6f4b5ce49ce2e062b6b36eb895259e17403"} Jul 29 15:47:06.084563 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:06.084538 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l7g6v" event={"ID":"846b9e1c-7ded-4a38-86ad-61f753398cf6","Type":"ContainerStarted","Data":"362f18451142ca88685d6d56e6b84ab618fad7966ccf279d6ff8e63250c8c46e"} Jul 29 15:47:07.094164 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:07.093904 2555 generic.go:356] "Generic (PLEG): container finished" podID="8341f0c8-f991-4c19-a797-f7fce085608f" containerID="e54a9def8d82961dd979d2554284585bf7ad0733e1c7bb86bb8f5e06508a5c5b" exitCode=0 Jul 29 15:47:07.094643 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:07.093955 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerDied","Data":"e54a9def8d82961dd979d2554284585bf7ad0733e1c7bb86bb8f5e06508a5c5b"} Jul 29 15:47:08.030862 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:08.030819 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:08.031017 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:08.030901 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:08.031017 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:08.031004 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:08.031132 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:08.031072 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls podName:571cee9d-30cc-4291-91c0-abb9ef90f3c5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:12.031057132 +0000 UTC m=+42.571958583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls") pod "dns-default-xmzml" (UID: "571cee9d-30cc-4291-91c0-abb9ef90f3c5") : secret "dns-default-metrics-tls" not found Jul 29 15:47:08.031132 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:08.031006 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 15:47:08.031132 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:08.031114 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768645b995-46b4w: secret "image-registry-tls" not found Jul 29 15:47:08.031227 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:08.031149 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls podName:4de6cfda-e440-4046-87eb-2f0ca0c54470 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:12.031137656 +0000 UTC m=+42.572039108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls") pod "image-registry-768645b995-46b4w" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470") : secret "image-registry-tls" not found Jul 29 15:47:08.100149 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:08.100106 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" event={"ID":"8341f0c8-f991-4c19-a797-f7fce085608f","Type":"ContainerStarted","Data":"42cd99a59dc9959467a588965f2cae1c1c6f9d045ed56e944ca4b75a648719d3"} Jul 29 15:47:08.101519 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:08.101492 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l7g6v" event={"ID":"846b9e1c-7ded-4a38-86ad-61f753398cf6","Type":"ContainerStarted","Data":"371706df06affae721ed62ee5b2c25ff4e981b93d7e030cb5ae6edafcc641dc6"} Jul 29 15:47:08.130772 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:08.130717 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8pzdz" podStartSLOduration=6.902710668 podStartE2EDuration="38.130698916s" totalImagesPullingTime="31.227988248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:08.129972692 +0000 UTC m=+38.670874155" watchObservedRunningTime="2026-07-29 15:47:08.130698916 +0000 UTC m=+38.671600384" Jul 29 15:47:08.146290 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:08.146242 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l7g6v" podStartSLOduration=1.7793314470000001 podStartE2EDuration="4.146226021s" totalImagesPullingTime="2.366894574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:08.145489791 +0000 UTC m=+38.686391260" watchObservedRunningTime="2026-07-29 15:47:08.146226021 +0000 UTC m=+38.687127491" Jul 29 15:47:09.785321 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.785288 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc"] Jul 29 15:47:09.806396 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.806359 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc"] Jul 29 15:47:09.806560 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.806481 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" Jul 29 15:47:09.809521 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.809499 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:47:09.809761 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.809748 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-htlkf\"" Jul 29 15:47:09.810746 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.810727 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 29 15:47:09.884154 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.884122 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv"] Jul 29 15:47:09.900295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.900254 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp"] Jul 29 15:47:09.900454 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.900373 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" Jul 29 15:47:09.903253 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.903229 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mzwpm\"" Jul 29 15:47:09.922596 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.922547 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n"] Jul 29 15:47:09.922752 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.922642 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:09.925199 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.925157 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 29 15:47:09.925199 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.925166 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jmrmv\"" Jul 29 15:47:09.925438 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.925410 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:47:09.925503 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.925478 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 29 15:47:09.925503 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.925417 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 29 15:47:09.944486 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.944452 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz"] Jul 29 15:47:09.944692 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.944663 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:09.944826 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.944797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nss2n\" (UniqueName: \"kubernetes.io/projected/8d1827c0-2505-404a-8af6-e386069efca2-kube-api-access-nss2n\") pod \"volume-data-source-validator-549c6d54cf-2fhqc\" (UID: \"8d1827c0-2505-404a-8af6-e386069efca2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" Jul 29 15:47:09.947490 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.947466 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2l9p6\"" Jul 29 15:47:09.947638 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.947466 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 15:47:09.948514 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.948491 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 29 15:47:09.948657 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.948560 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 29 15:47:09.948657 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.948634 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 15:47:09.966675 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.966639 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-658f857f8f-j4knh"] Jul 29 15:47:09.966811 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.966755 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:09.969412 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.969387 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jwqzs\"" Jul 29 15:47:09.969564 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.969462 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:47:09.969564 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.969539 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 15:47:09.969694 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.969682 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 15:47:09.989100 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.988760 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv"] Jul 29 15:47:09.989277 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.989233 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:09.991775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.991752 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 15:47:09.993174 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.993150 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 15:47:09.993420 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.993333 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 15:47:09.993503 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.993447 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 15:47:09.993503 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.993457 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 15:47:09.993635 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.993623 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 15:47:09.993894 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.993872 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dldjd\"" Jul 29 15:47:09.994037 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.994017 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n"] Jul 29 15:47:09.994148 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.994069 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp"] Jul 29 15:47:09.994148 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.994103 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz"] Jul 29 15:47:09.994148 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.994139 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-658f857f8f-j4knh"] Jul 29 15:47:09.994301 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:09.994181 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq"] Jul 29 15:47:10.010969 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.010937 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-vrsmf"] Jul 29 15:47:10.011152 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.011065 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.013793 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.013770 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:47:10.013924 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.013821 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 15:47:10.014078 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.014063 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 15:47:10.014124 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.014111 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 15:47:10.014210 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.014192 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gh2cx\"" Jul 29 15:47:10.024792 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.024771 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-6mrvv"] Jul 29 15:47:10.024939 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.024923 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.030312 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.030165 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 15:47:10.030312 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.030256 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 29 15:47:10.030647 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.030616 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 29 15:47:10.030945 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.030931 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 15:47:10.031184 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.031168 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wrzv9\"" Jul 29 15:47:10.036085 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.036024 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 29 15:47:10.040958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.040938 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq"] Jul 29 15:47:10.041071 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.040963 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-vrsmf"] Jul 29 15:47:10.041071 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.040977 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-6mrvv"] Jul 29 15:47:10.041071 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.041044 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.043451 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.043434 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8zgb9\"" Jul 29 15:47:10.043560 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.043509 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 29 15:47:10.044281 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.044264 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 29 15:47:10.044449 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.044424 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:47:10.044828 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.044813 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 29 15:47:10.045226 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045204 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cabeca35-f4d0-4626-82d2-be6ded895a50-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.045297 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.045297 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045284 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhn9\" (UniqueName: \"kubernetes.io/projected/cabeca35-f4d0-4626-82d2-be6ded895a50-kube-api-access-hrhn9\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.045395 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045311 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:10.045395 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045337 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2w8\" (UniqueName: \"kubernetes.io/projected/035aae76-c47c-4476-aca7-792b6bb51dce-kube-api-access-hb2w8\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:10.045395 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045383 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68gsw\" (UniqueName: \"kubernetes.io/projected/b18913da-9faf-4eb2-bfbf-f63e9308022e-kube-api-access-68gsw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.045511 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045414 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b18913da-9faf-4eb2-bfbf-f63e9308022e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.045511 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045443 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kfb\" (UniqueName: \"kubernetes.io/projected/f78b8654-f388-4a88-ba88-375e684e344a-kube-api-access-b9kfb\") pod \"network-check-source-7d88cb55d8-ssxfv\" (UID: \"f78b8654-f388-4a88-ba88-375e684e344a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" Jul 29 15:47:10.045619 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045537 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nss2n\" (UniqueName: \"kubernetes.io/projected/8d1827c0-2505-404a-8af6-e386069efca2-kube-api-access-nss2n\") pod \"volume-data-source-validator-549c6d54cf-2fhqc\" (UID: \"8d1827c0-2505-404a-8af6-e386069efca2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" Jul 29 15:47:10.045619 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.045561 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b18913da-9faf-4eb2-bfbf-f63e9308022e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.049926 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.049908 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 29 15:47:10.057854 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.057832 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nss2n\" (UniqueName: \"kubernetes.io/projected/8d1827c0-2505-404a-8af6-e386069efca2-kube-api-access-nss2n\") pod \"volume-data-source-validator-549c6d54cf-2fhqc\" (UID: \"8d1827c0-2505-404a-8af6-e386069efca2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" Jul 29 15:47:10.115713 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.115670 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" Jul 29 15:47:10.146982 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.146947 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxf2\" (UniqueName: \"kubernetes.io/projected/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-kube-api-access-dhxf2\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.147126 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.146994 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kfb\" (UniqueName: \"kubernetes.io/projected/f78b8654-f388-4a88-ba88-375e684e344a-kube-api-access-b9kfb\") pod \"network-check-source-7d88cb55d8-ssxfv\" (UID: \"f78b8654-f388-4a88-ba88-375e684e344a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" Jul 29 15:47:10.147126 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147015 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-config\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.147126 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147057 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-stats-auth\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.147126 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147073 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-snapshots\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.147126 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147097 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b18913da-9faf-4eb2-bfbf-f63e9308022e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.147126 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147126 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9bs\" (UniqueName: \"kubernetes.io/projected/1447efed-1b87-414e-a949-d0db6b807f42-kube-api-access-lz9bs\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.147416 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147292 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h7pp\" (UniqueName: \"kubernetes.io/projected/edc2889a-6586-468b-9fa8-f7d8ff946770-kube-api-access-4h7pp\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.147416 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147323 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhn9\" (UniqueName: \"kubernetes.io/projected/cabeca35-f4d0-4626-82d2-be6ded895a50-kube-api-access-hrhn9\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.147416 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147350 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1447efed-1b87-414e-a949-d0db6b807f42-config\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.147416 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147374 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:10.147416 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147401 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2w8\" (UniqueName: \"kubernetes.io/projected/035aae76-c47c-4476-aca7-792b6bb51dce-kube-api-access-hb2w8\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:10.147685 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147443 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-service-ca-bundle\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.147685 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147470 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.147685 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.147685 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.147525 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 15:47:10.147685 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147563 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-trusted-ca\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.147883 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147724 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gsw\" (UniqueName: \"kubernetes.io/projected/b18913da-9faf-4eb2-bfbf-f63e9308022e-kube-api-access-68gsw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.147883 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.147764 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls podName:035aae76-c47c-4476-aca7-792b6bb51dce nodeName:}" failed. No retries permitted until 2026-07-29 15:47:10.647731502 +0000 UTC m=+41.188632949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls") pod "cluster-samples-operator-6748489656-h4vdz" (UID: "035aae76-c47c-4476-aca7-792b6bb51dce") : secret "samples-operator-tls" not found Jul 29 15:47:10.147883 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.147771 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:10.147883 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147789 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b18913da-9faf-4eb2-bfbf-f63e9308022e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.147883 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147840 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.148055 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.147884 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls podName:cabeca35-f4d0-4626-82d2-be6ded895a50 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:10.647863254 +0000 UTC m=+41.188764701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8hm8n" (UID: "cabeca35-f4d0-4626-82d2-be6ded895a50") : secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:10.148055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147931 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b18913da-9faf-4eb2-bfbf-f63e9308022e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.148055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.147978 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-serving-cert\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.148055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148007 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1447efed-1b87-414e-a949-d0db6b807f42-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.148189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148059 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-tmp\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.148189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148097 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.148189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148125 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cabeca35-f4d0-4626-82d2-be6ded895a50-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.148189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148153 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-default-certificate\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.148189 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148182 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7dr\" (UniqueName: \"kubernetes.io/projected/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-kube-api-access-hm7dr\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.148414 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148217 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-serving-cert\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.148839 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.148821 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cabeca35-f4d0-4626-82d2-be6ded895a50-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.150107 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.150090 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b18913da-9faf-4eb2-bfbf-f63e9308022e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.156662 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.156630 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gsw\" (UniqueName: \"kubernetes.io/projected/b18913da-9faf-4eb2-bfbf-f63e9308022e-kube-api-access-68gsw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp\" (UID: \"b18913da-9faf-4eb2-bfbf-f63e9308022e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.157215 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.157192 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhn9\" (UniqueName: \"kubernetes.io/projected/cabeca35-f4d0-4626-82d2-be6ded895a50-kube-api-access-hrhn9\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.157481 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.157450 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2w8\" (UniqueName: \"kubernetes.io/projected/035aae76-c47c-4476-aca7-792b6bb51dce-kube-api-access-hb2w8\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:10.158570 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.158554 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kfb\" (UniqueName: \"kubernetes.io/projected/f78b8654-f388-4a88-ba88-375e684e344a-kube-api-access-b9kfb\") pod \"network-check-source-7d88cb55d8-ssxfv\" (UID: \"f78b8654-f388-4a88-ba88-375e684e344a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" Jul 29 15:47:10.210555 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.210519 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" Jul 29 15:47:10.233958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.233928 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" Jul 29 15:47:10.249468 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249436 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9bs\" (UniqueName: \"kubernetes.io/projected/1447efed-1b87-414e-a949-d0db6b807f42-kube-api-access-lz9bs\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.249602 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249506 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h7pp\" (UniqueName: \"kubernetes.io/projected/edc2889a-6586-468b-9fa8-f7d8ff946770-kube-api-access-4h7pp\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.249602 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1447efed-1b87-414e-a949-d0db6b807f42-config\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.249602 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-service-ca-bundle\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.249741 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249630 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.249741 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249657 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-trusted-ca\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.249741 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.249905 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.249769 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 15:47:10.249905 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.249843 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:10.749821992 +0000 UTC m=+41.290723453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : secret "router-metrics-certs-default" not found Jul 29 15:47:10.249905 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249888 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-serving-cert\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.250072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249922 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1447efed-1b87-414e-a949-d0db6b807f42-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.250072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.249972 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-tmp\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.250072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250015 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.250072 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250053 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-default-certificate\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7dr\" (UniqueName: \"kubernetes.io/projected/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-kube-api-access-hm7dr\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250117 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-serving-cert\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250153 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxf2\" (UniqueName: \"kubernetes.io/projected/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-kube-api-access-dhxf2\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250181 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-config\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250220 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-stats-auth\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250245 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-snapshots\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250266 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-service-ca-bundle\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250277 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1447efed-1b87-414e-a949-d0db6b807f42-config\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.250403 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.250391 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:10.750373652 +0000 UTC m=+41.291275124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : configmap references non-existent config key: service-ca.crt Jul 29 15:47:10.250909 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250567 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-trusted-ca\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.250909 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.250834 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.251080 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.251055 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-snapshots\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.251719 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.251694 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-config\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.251883 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.251833 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-tmp\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.253063 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.253037 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-serving-cert\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.253416 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.253389 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-serving-cert\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.253511 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.253417 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-stats-auth\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.253573 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.253543 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-default-certificate\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.253654 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.253610 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1447efed-1b87-414e-a949-d0db6b807f42-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.257892 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.257866 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9bs\" (UniqueName: \"kubernetes.io/projected/1447efed-1b87-414e-a949-d0db6b807f42-kube-api-access-lz9bs\") pod \"service-ca-operator-5d6c75f84f-wmqwq\" (UID: \"1447efed-1b87-414e-a949-d0db6b807f42\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.258517 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.258495 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h7pp\" (UniqueName: \"kubernetes.io/projected/edc2889a-6586-468b-9fa8-f7d8ff946770-kube-api-access-4h7pp\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.260118 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.260099 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7dr\" (UniqueName: \"kubernetes.io/projected/d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec-kube-api-access-hm7dr\") pod \"console-operator-584dcb6b54-6mrvv\" (UID: \"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec\") " pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.260418 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.260396 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxf2\" (UniqueName: \"kubernetes.io/projected/cba9d9bd-cb1c-48e0-a19d-46c76e955a4f-kube-api-access-dhxf2\") pod \"insights-operator-5885fdb759-vrsmf\" (UID: \"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f\") " pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.267627 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.267608 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc"] Jul 29 15:47:10.269679 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:10.269637 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1827c0_2505_404a_8af6_e386069efca2.slice/crio-fa46400bfcda6b693e34dea490572215aec7f8af4dcec1b7e9af6305b86df063 WatchSource:0}: Error finding container fa46400bfcda6b693e34dea490572215aec7f8af4dcec1b7e9af6305b86df063: Status 404 returned error can't find the container with id fa46400bfcda6b693e34dea490572215aec7f8af4dcec1b7e9af6305b86df063 Jul 29 15:47:10.321974 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.321711 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" Jul 29 15:47:10.334081 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.334010 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-vrsmf" Jul 29 15:47:10.350282 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.350190 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:10.422380 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.419393 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv"] Jul 29 15:47:10.448816 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.447976 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp"] Jul 29 15:47:10.451018 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:10.450959 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18913da_9faf_4eb2_bfbf_f63e9308022e.slice/crio-72c426edff89150eb859ab846d5a585039b7ccac7598c386da285256daf3bebf WatchSource:0}: Error finding container 72c426edff89150eb859ab846d5a585039b7ccac7598c386da285256daf3bebf: Status 404 returned error can't find the container with id 72c426edff89150eb859ab846d5a585039b7ccac7598c386da285256daf3bebf Jul 29 15:47:10.508335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.508289 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq"] Jul 29 15:47:10.512041 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:10.512008 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1447efed_1b87_414e_a949_d0db6b807f42.slice/crio-54ff348e73fc42ffd26afb0fef777c168f9ea19b419e394356e698fa32472f5b WatchSource:0}: Error finding container 54ff348e73fc42ffd26afb0fef777c168f9ea19b419e394356e698fa32472f5b: Status 404 returned error can't find the container with id 54ff348e73fc42ffd26afb0fef777c168f9ea19b419e394356e698fa32472f5b Jul 29 15:47:10.529978 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.529948 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-vrsmf"] Jul 29 15:47:10.532573 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:10.532542 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba9d9bd_cb1c_48e0_a19d_46c76e955a4f.slice/crio-05dd31613adf643df1b6817e6a142577133673d7c7452cecf91aeaa0b2862a88 WatchSource:0}: Error finding container 05dd31613adf643df1b6817e6a142577133673d7c7452cecf91aeaa0b2862a88: Status 404 returned error can't find the container with id 05dd31613adf643df1b6817e6a142577133673d7c7452cecf91aeaa0b2862a88 Jul 29 15:47:10.554821 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.554741 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-6mrvv"] Jul 29 15:47:10.558173 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:10.558149 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e2ec2f_3b47_4eb1_99b4_eb44bc7e5bec.slice/crio-153023520ae7f430fb3299fdd5745e5cbc56a2fb0ac4f3eb69d69228e2266ae2 WatchSource:0}: Error finding container 153023520ae7f430fb3299fdd5745e5cbc56a2fb0ac4f3eb69d69228e2266ae2: Status 404 returned error can't find the container with id 153023520ae7f430fb3299fdd5745e5cbc56a2fb0ac4f3eb69d69228e2266ae2 Jul 29 15:47:10.653782 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.653742 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:10.653782 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.653787 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:10.653973 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.653870 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:10.653973 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.653878 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 15:47:10.653973 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.653924 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls podName:cabeca35-f4d0-4626-82d2-be6ded895a50 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:11.653909387 +0000 UTC m=+42.194810834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8hm8n" (UID: "cabeca35-f4d0-4626-82d2-be6ded895a50") : secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:10.653973 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.653969 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls podName:035aae76-c47c-4476-aca7-792b6bb51dce nodeName:}" failed. No retries permitted until 2026-07-29 15:47:11.653963363 +0000 UTC m=+42.194864810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls") pod "cluster-samples-operator-6748489656-h4vdz" (UID: "035aae76-c47c-4476-aca7-792b6bb51dce") : secret "samples-operator-tls" not found Jul 29 15:47:10.754652 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.754608 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.754849 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:10.754721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:10.754849 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.754799 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:11.754778208 +0000 UTC m=+42.295679656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : configmap references non-existent config key: service-ca.crt Jul 29 15:47:10.754849 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.754817 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 15:47:10.754969 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:10.754856 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:11.754845009 +0000 UTC m=+42.295746456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : secret "router-metrics-certs-default" not found Jul 29 15:47:11.113952 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.113861 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vrsmf" event={"ID":"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f","Type":"ContainerStarted","Data":"05dd31613adf643df1b6817e6a142577133673d7c7452cecf91aeaa0b2862a88"} Jul 29 15:47:11.117610 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.117531 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" event={"ID":"8d1827c0-2505-404a-8af6-e386069efca2","Type":"ContainerStarted","Data":"fa46400bfcda6b693e34dea490572215aec7f8af4dcec1b7e9af6305b86df063"} Jul 29 15:47:11.119339 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.119305 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" event={"ID":"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec","Type":"ContainerStarted","Data":"153023520ae7f430fb3299fdd5745e5cbc56a2fb0ac4f3eb69d69228e2266ae2"} Jul 29 15:47:11.121325 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.121279 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" event={"ID":"1447efed-1b87-414e-a949-d0db6b807f42","Type":"ContainerStarted","Data":"54ff348e73fc42ffd26afb0fef777c168f9ea19b419e394356e698fa32472f5b"} Jul 29 15:47:11.122914 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.122884 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" event={"ID":"b18913da-9faf-4eb2-bfbf-f63e9308022e","Type":"ContainerStarted","Data":"72c426edff89150eb859ab846d5a585039b7ccac7598c386da285256daf3bebf"} Jul 29 15:47:11.124299 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.124273 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" event={"ID":"f78b8654-f388-4a88-ba88-375e684e344a","Type":"ContainerStarted","Data":"be317e0a712d8aa4be2d21e40f7ef21b2e7e8d2aa99885313cdcdc32a07de5f3"} Jul 29 15:47:11.665802 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.665765 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:11.665958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.665831 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:11.666022 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:11.666008 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:11.666085 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:11.666074 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls podName:cabeca35-f4d0-4626-82d2-be6ded895a50 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:13.666053996 +0000 UTC m=+44.206955447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8hm8n" (UID: "cabeca35-f4d0-4626-82d2-be6ded895a50") : secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:11.666158 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:11.666146 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 15:47:11.666217 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:11.666184 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls podName:035aae76-c47c-4476-aca7-792b6bb51dce nodeName:}" failed. No retries permitted until 2026-07-29 15:47:13.666171297 +0000 UTC m=+44.207072759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls") pod "cluster-samples-operator-6748489656-h4vdz" (UID: "035aae76-c47c-4476-aca7-792b6bb51dce") : secret "samples-operator-tls" not found Jul 29 15:47:11.766886 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.766845 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:11.767055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:11.766954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:11.767134 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:11.767098 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:13.767077652 +0000 UTC m=+44.307979105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : configmap references non-existent config key: service-ca.crt Jul 29 15:47:11.767526 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:11.767504 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 15:47:11.767631 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:11.767561 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:13.767546974 +0000 UTC m=+44.308448426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : secret "router-metrics-certs-default" not found Jul 29 15:47:12.069901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:12.069859 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:12.070097 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:12.069959 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:12.070270 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:12.070232 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 15:47:12.070270 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:12.070250 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768645b995-46b4w: secret "image-registry-tls" not found Jul 29 15:47:12.070376 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:12.070311 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls podName:4de6cfda-e440-4046-87eb-2f0ca0c54470 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:20.070290091 +0000 UTC m=+50.611191541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls") pod "image-registry-768645b995-46b4w" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470") : secret "image-registry-tls" not found Jul 29 15:47:12.070781 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:12.070759 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:12.070890 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:12.070820 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls podName:571cee9d-30cc-4291-91c0-abb9ef90f3c5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:20.070803461 +0000 UTC m=+50.611704913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls") pod "dns-default-xmzml" (UID: "571cee9d-30cc-4291-91c0-abb9ef90f3c5") : secret "dns-default-metrics-tls" not found Jul 29 15:47:13.684895 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:13.684854 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:13.685310 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:13.684907 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:13.685310 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:13.685013 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 15:47:13.685310 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:13.685045 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:13.685310 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:13.685079 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls podName:035aae76-c47c-4476-aca7-792b6bb51dce nodeName:}" failed. No retries permitted until 2026-07-29 15:47:17.685064211 +0000 UTC m=+48.225965657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls") pod "cluster-samples-operator-6748489656-h4vdz" (UID: "035aae76-c47c-4476-aca7-792b6bb51dce") : secret "samples-operator-tls" not found Jul 29 15:47:13.685310 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:13.685094 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls podName:cabeca35-f4d0-4626-82d2-be6ded895a50 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:17.685087324 +0000 UTC m=+48.225988771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8hm8n" (UID: "cabeca35-f4d0-4626-82d2-be6ded895a50") : secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:13.785628 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:13.785560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:13.785790 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:13.785679 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:13.785790 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:13.785704 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 15:47:13.785790 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:13.785775 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:17.785756479 +0000 UTC m=+48.326657951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : secret "router-metrics-certs-default" not found Jul 29 15:47:13.785896 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:13.785794 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:17.785783941 +0000 UTC m=+48.326685388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : configmap references non-existent config key: service-ca.crt Jul 29 15:47:17.148194 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.147924 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" event={"ID":"f78b8654-f388-4a88-ba88-375e684e344a","Type":"ContainerStarted","Data":"3cb74bd6b8c52e3e03dcd48de226577ec46202662c82fc552868605989d5d70b"} Jul 29 15:47:17.149737 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.149682 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vrsmf" event={"ID":"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f","Type":"ContainerStarted","Data":"f0267e4c69a8c5599e1591a46f38493d332d09e22a43b32cce12485142309415"} Jul 29 15:47:17.151370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.151345 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" event={"ID":"8d1827c0-2505-404a-8af6-e386069efca2","Type":"ContainerStarted","Data":"636b11f886b2e2524e28d7db5501216904aae74d72e0b6cb121118a235f35a85"} Jul 29 15:47:17.153816 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.153619 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" event={"ID":"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec","Type":"ContainerStarted","Data":"cf538ecbdfa229e003c3f50a3d37d0b6e3919e6fb3518139e649e992410800f4"} Jul 29 15:47:17.154281 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.154001 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:17.155850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.155618 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" event={"ID":"1447efed-1b87-414e-a949-d0db6b807f42","Type":"ContainerStarted","Data":"9967afad9e8bd70076ba8ab96d042560f8b29f3bd7c8aa8d598aeb389d5895ef"} Jul 29 15:47:17.156211 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.156162 2555 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-6mrvv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Jul 29 15:47:17.156323 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.156210 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" podUID="d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Jul 29 15:47:17.157538 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.157513 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" event={"ID":"b18913da-9faf-4eb2-bfbf-f63e9308022e","Type":"ContainerStarted","Data":"036b60279819c836d76f330138c454317acf450281372414f45cf0063b28c3c0"} Jul 29 15:47:17.171923 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.171862 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ssxfv" podStartSLOduration=1.804673527 podStartE2EDuration="8.171840516s" totalImagesPullingTime="6.367166989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:17.166132715 +0000 UTC m=+47.707034164" watchObservedRunningTime="2026-07-29 15:47:17.171840516 +0000 UTC m=+47.712741967" Jul 29 15:47:17.185122 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.185056 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2fhqc" podStartSLOduration=1.775002883 podStartE2EDuration="8.185032786s" totalImagesPullingTime="6.410029903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:17.183639702 +0000 UTC m=+47.724541168" watchObservedRunningTime="2026-07-29 15:47:17.185032786 +0000 UTC m=+47.725934256" Jul 29 15:47:17.209331 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.209264 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" podStartSLOduration=1.935157668 podStartE2EDuration="8.209240378s" totalImagesPullingTime="6.27408271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:17.207446403 +0000 UTC m=+47.748347874" watchObservedRunningTime="2026-07-29 15:47:17.209240378 +0000 UTC m=+47.750141849" Jul 29 15:47:17.232688 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.232626 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-vrsmf" podStartSLOduration=1.978720727 podStartE2EDuration="8.232575332s" totalImagesPullingTime="6.253854605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:17.231262307 +0000 UTC m=+47.772163776" watchObservedRunningTime="2026-07-29 15:47:17.232575332 +0000 UTC m=+47.773476802" Jul 29 15:47:17.257162 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.257094 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" podStartSLOduration=2.022439349 podStartE2EDuration="8.257060724s" totalImagesPullingTime="6.234621375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:17.256500827 +0000 UTC m=+47.797402299" watchObservedRunningTime="2026-07-29 15:47:17.257060724 +0000 UTC m=+47.797962194" Jul 29 15:47:17.280029 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.279964 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" podStartSLOduration=1.945074113 podStartE2EDuration="8.2799391s" totalImagesPullingTime="6.334864987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:17.27865207 +0000 UTC m=+47.819553538" watchObservedRunningTime="2026-07-29 15:47:17.2799391 +0000 UTC m=+47.820840570" Jul 29 15:47:17.722101 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.722052 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:17.722313 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:17.722229 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:17.722313 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.722267 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:17.722437 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:17.722314 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls podName:cabeca35-f4d0-4626-82d2-be6ded895a50 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:25.722293344 +0000 UTC m=+56.263194804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8hm8n" (UID: "cabeca35-f4d0-4626-82d2-be6ded895a50") : secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:17.722437 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:17.722421 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 15:47:17.722548 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:17.722482 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls podName:035aae76-c47c-4476-aca7-792b6bb51dce nodeName:}" failed. No retries permitted until 2026-07-29 15:47:25.722466028 +0000 UTC m=+56.263367481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls") pod "cluster-samples-operator-6748489656-h4vdz" (UID: "035aae76-c47c-4476-aca7-792b6bb51dce") : secret "samples-operator-tls" not found Jul 29 15:47:17.823002 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.822961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:17.823190 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:17.823078 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:17.823190 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:17.823136 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 15:47:17.823307 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:17.823219 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:25.823196612 +0000 UTC m=+56.364098063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : secret "router-metrics-certs-default" not found Jul 29 15:47:17.823307 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:17.823246 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:25.823226122 +0000 UTC m=+56.364127586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : configmap references non-existent config key: service-ca.crt Jul 29 15:47:18.163202 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.163160 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/0.log" Jul 29 15:47:18.163832 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.163203 2555 generic.go:356] "Generic (PLEG): container finished" podID="d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec" containerID="cf538ecbdfa229e003c3f50a3d37d0b6e3919e6fb3518139e649e992410800f4" exitCode=255 Jul 29 15:47:18.163832 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.163235 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" event={"ID":"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec","Type":"ContainerDied","Data":"cf538ecbdfa229e003c3f50a3d37d0b6e3919e6fb3518139e649e992410800f4"} Jul 29 15:47:18.164145 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.164122 2555 scope.go:117] "RemoveContainer" containerID="cf538ecbdfa229e003c3f50a3d37d0b6e3919e6fb3518139e649e992410800f4" Jul 29 15:47:18.787653 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.787619 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p4n8x"] Jul 29 15:47:18.809796 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.809764 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p4n8x"] Jul 29 15:47:18.809939 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.809881 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.816814 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.816789 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 15:47:18.816814 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.816806 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sv279\"" Jul 29 15:47:18.816814 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.816790 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 15:47:18.834656 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.834613 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/794e5299-4224-4873-9c41-6857ad5c2390-crio-socket\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.834656 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.834657 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/794e5299-4224-4873-9c41-6857ad5c2390-data-volume\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.834927 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.834746 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.834927 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.834778 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzpd\" (UniqueName: \"kubernetes.io/projected/794e5299-4224-4873-9c41-6857ad5c2390-kube-api-access-dpzpd\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.834927 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.834848 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.935270 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.935230 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/794e5299-4224-4873-9c41-6857ad5c2390-crio-socket\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.935270 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.935270 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/794e5299-4224-4873-9c41-6857ad5c2390-data-volume\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.935541 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.935301 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.935541 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.935359 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/794e5299-4224-4873-9c41-6857ad5c2390-crio-socket\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.935541 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:18.935442 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 15:47:18.935541 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.935472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzpd\" (UniqueName: \"kubernetes.io/projected/794e5299-4224-4873-9c41-6857ad5c2390-kube-api-access-dpzpd\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.935541 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:18.935505 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:19.435485722 +0000 UTC m=+49.976387169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : configmap "kube-rbac-proxy" not found Jul 29 15:47:18.935864 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.935550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.935864 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:18.935673 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:18.935864 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:18.935731 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:19.435718245 +0000 UTC m=+49.976619711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:18.935864 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.935671 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/794e5299-4224-4873-9c41-6857ad5c2390-data-volume\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:18.950708 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:18.950676 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzpd\" (UniqueName: \"kubernetes.io/projected/794e5299-4224-4873-9c41-6857ad5c2390-kube-api-access-dpzpd\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:19.168170 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.168097 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 15:47:19.168536 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.168486 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/0.log" Jul 29 15:47:19.168536 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.168518 2555 generic.go:356] "Generic (PLEG): container finished" podID="d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec" containerID="776d5c3f8127ed9dd6bed028a55926cbd1ede0a7d562f316b3a39897818c39b7" exitCode=255 Jul 29 15:47:19.168624 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.168549 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" event={"ID":"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec","Type":"ContainerDied","Data":"776d5c3f8127ed9dd6bed028a55926cbd1ede0a7d562f316b3a39897818c39b7"} Jul 29 15:47:19.168660 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.168631 2555 scope.go:117] "RemoveContainer" containerID="cf538ecbdfa229e003c3f50a3d37d0b6e3919e6fb3518139e649e992410800f4" Jul 29 15:47:19.168902 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.168874 2555 scope.go:117] "RemoveContainer" containerID="776d5c3f8127ed9dd6bed028a55926cbd1ede0a7d562f316b3a39897818c39b7" Jul 29 15:47:19.169095 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:19.169064 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-6mrvv_openshift-console-operator(d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec)\"" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" podUID="d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec" Jul 29 15:47:19.440218 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.440130 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:19.440390 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:19.440315 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:19.440463 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:19.440393 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:20.44037175 +0000 UTC m=+50.981273201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:19.440463 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:19.440434 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:19.440578 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:19.440533 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 15:47:19.440656 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:19.440606 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:20.440568424 +0000 UTC m=+50.981469886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : configmap "kube-rbac-proxy" not found Jul 29 15:47:20.147989 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.147942 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:20.148171 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.148043 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:20.148171 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.148093 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:20.148171 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.148166 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls podName:571cee9d-30cc-4291-91c0-abb9ef90f3c5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:36.148149165 +0000 UTC m=+66.689050615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls") pod "dns-default-xmzml" (UID: "571cee9d-30cc-4291-91c0-abb9ef90f3c5") : secret "dns-default-metrics-tls" not found Jul 29 15:47:20.148279 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.148198 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 15:47:20.148279 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.148215 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768645b995-46b4w: secret "image-registry-tls" not found Jul 29 15:47:20.148279 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.148269 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls podName:4de6cfda-e440-4046-87eb-2f0ca0c54470 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:36.148256093 +0000 UTC m=+66.689157539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls") pod "image-registry-768645b995-46b4w" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470") : secret "image-registry-tls" not found Jul 29 15:47:20.173107 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.173081 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 15:47:20.173491 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.173471 2555 scope.go:117] "RemoveContainer" containerID="776d5c3f8127ed9dd6bed028a55926cbd1ede0a7d562f316b3a39897818c39b7" Jul 29 15:47:20.173720 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.173697 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-6mrvv_openshift-console-operator(d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec)\"" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" podUID="d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec" Jul 29 15:47:20.350710 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.350674 2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:20.451371 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.451273 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:20.451371 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.451346 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:20.451604 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.451411 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 15:47:20.451604 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.451484 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:20.451604 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.451502 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:22.451481915 +0000 UTC m=+52.992383368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : configmap "kube-rbac-proxy" not found Jul 29 15:47:20.451604 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:20.451542 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:22.451529246 +0000 UTC m=+52.992430692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:20.976431 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:20.976396 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m7jv2_be264cf7-2ed3-4c99-8a85-264cacb645c7/dns-node-resolver/0.log" Jul 29 15:47:21.176959 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:21.176935 2555 scope.go:117] "RemoveContainer" containerID="776d5c3f8127ed9dd6bed028a55926cbd1ede0a7d562f316b3a39897818c39b7" Jul 29 15:47:21.177341 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:21.177093 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-6mrvv_openshift-console-operator(d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec)\"" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" podUID="d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec" Jul 29 15:47:21.977726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:21.977698 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6mvd9_120f7d41-d050-48d7-acd8-c9779c7af52f/node-ca/0.log" Jul 29 15:47:22.469538 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:22.469493 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:22.469958 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:22.469550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:22.469958 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:22.469700 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:22.469958 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:22.469762 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:26.469744282 +0000 UTC m=+57.010645751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:22.470059 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:22.470045 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/794e5299-4224-4873-9c41-6857ad5c2390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:22.576374 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:22.576345 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l7g6v_846b9e1c-7ded-4a38-86ad-61f753398cf6/serve-healthcheck-canary/0.log" Jul 29 15:47:25.798071 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:25.798026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:25.798071 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:25.798080 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:25.798677 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:25.798223 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:25.798677 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:25.798319 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls podName:cabeca35-f4d0-4626-82d2-be6ded895a50 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:41.798297097 +0000 UTC m=+72.339198567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8hm8n" (UID: "cabeca35-f4d0-4626-82d2-be6ded895a50") : secret "cluster-monitoring-operator-tls" not found Jul 29 15:47:25.800484 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:25.800460 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/035aae76-c47c-4476-aca7-792b6bb51dce-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-h4vdz\" (UID: \"035aae76-c47c-4476-aca7-792b6bb51dce\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:25.875798 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:25.875758 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" Jul 29 15:47:25.898906 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:25.898874 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:25.899080 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:25.898954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:25.899080 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:25.899018 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 15:47:25.899195 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:25.899098 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:41.899076382 +0000 UTC m=+72.439977843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : secret "router-metrics-certs-default" not found Jul 29 15:47:25.899195 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:25.899118 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle podName:edc2889a-6586-468b-9fa8-f7d8ff946770 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:41.899108748 +0000 UTC m=+72.440010195 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle") pod "router-default-658f857f8f-j4knh" (UID: "edc2889a-6586-468b-9fa8-f7d8ff946770") : configmap references non-existent config key: service-ca.crt Jul 29 15:47:26.010279 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:26.010244 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz"] Jul 29 15:47:26.192675 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:26.192634 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" event={"ID":"035aae76-c47c-4476-aca7-792b6bb51dce","Type":"ContainerStarted","Data":"f35d276cb1693c32fd1aace1009116e93404ab49fa482e8b09dc001a21013346"} Jul 29 15:47:26.505046 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:26.504955 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:26.505224 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:26.505132 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:26.505289 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:26.505231 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls podName:794e5299-4224-4873-9c41-6857ad5c2390 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:34.505206825 +0000 UTC m=+65.046108286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4n8x" (UID: "794e5299-4224-4873-9c41-6857ad5c2390") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:27.154272 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:27.154230 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:27.154708 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:27.154675 2555 scope.go:117] "RemoveContainer" containerID="776d5c3f8127ed9dd6bed028a55926cbd1ede0a7d562f316b3a39897818c39b7" Jul 29 15:47:27.154852 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:47:27.154832 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-6mrvv_openshift-console-operator(d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec)\"" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" podUID="d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec" Jul 29 15:47:29.203815 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:29.203778 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" event={"ID":"035aae76-c47c-4476-aca7-792b6bb51dce","Type":"ContainerStarted","Data":"21d2d7bc8711cedca7cf663ceb07cbaf36e79613baaeae71eee432ba7b733848"} Jul 29 15:47:29.204220 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:29.203863 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" event={"ID":"035aae76-c47c-4476-aca7-792b6bb51dce","Type":"ContainerStarted","Data":"a52c96402471eb94b00b05fd652ce23d001c46e49b53abb390f47ef8776cbfc3"} Jul 29 15:47:30.072194 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:30.072167 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nqdlk" Jul 29 15:47:30.103401 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:30.103349 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-h4vdz" podStartSLOduration=18.908064163 podStartE2EDuration="21.103331593s" totalImagesPullingTime="2.19526743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:29.221909472 +0000 UTC m=+59.762810940" watchObservedRunningTime="2026-07-29 15:47:30.103331593 +0000 UTC m=+60.644233062" Jul 29 15:47:34.580453 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.580412 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:34.582862 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.582835 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/794e5299-4224-4873-9c41-6857ad5c2390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4n8x\" (UID: \"794e5299-4224-4873-9c41-6857ad5c2390\") " pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:34.722101 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.722068 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sv279\"" Jul 29 15:47:34.730312 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.730289 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p4n8x" Jul 29 15:47:34.782899 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.782401 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:34.782899 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.782455 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:47:34.782899 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.782547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:47:34.785752 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.785730 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 15:47:34.785914 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.785730 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 15:47:34.786015 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.785782 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjwl\" (UniqueName: \"kubernetes.io/projected/02e74657-8c54-477b-a184-eb2a35c8eff5-kube-api-access-2xjwl\") pod \"network-check-target-gqd46\" (UID: \"02e74657-8c54-477b-a184-eb2a35c8eff5\") " pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:34.796212 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.796172 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cffac7-1815-42f4-8bc7-ed8a8996b20f-metrics-certs\") pod \"network-metrics-daemon-29fg9\" (UID: \"69cffac7-1815-42f4-8bc7-ed8a8996b20f\") " pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:47:34.796633 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.796613 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6d0e693-ccf2-4168-a63c-2f1428af95fe-original-pull-secret\") pod \"global-pull-secret-syncer-ptml9\" (UID: \"c6d0e693-ccf2-4168-a63c-2f1428af95fe\") " pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:47:34.859991 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.859907 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p4n8x"] Jul 29 15:47:34.863762 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:34.863733 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794e5299_4224_4873_9c41_6857ad5c2390.slice/crio-4a37b9929214486af29cf719b900e119ad76082838f8a39053b1d8225a460002 WatchSource:0}: Error finding container 4a37b9929214486af29cf719b900e119ad76082838f8a39053b1d8225a460002: Status 404 returned error can't find the container with id 4a37b9929214486af29cf719b900e119ad76082838f8a39053b1d8225a460002 Jul 29 15:47:34.933156 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.933128 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28zhr\"" Jul 29 15:47:34.940465 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:34.940441 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29fg9" Jul 29 15:47:35.031910 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.031879 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptml9" Jul 29 15:47:35.039890 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.039860 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hf4f8\"" Jul 29 15:47:35.047545 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.047497 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:35.068116 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.068086 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29fg9"] Jul 29 15:47:35.071369 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:35.071341 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cffac7_1815_42f4_8bc7_ed8a8996b20f.slice/crio-2bb96c12c710c24ff38b1bb9d49f637786a37845a5a45f419b22edadde78c201 WatchSource:0}: Error finding container 2bb96c12c710c24ff38b1bb9d49f637786a37845a5a45f419b22edadde78c201: Status 404 returned error can't find the container with id 2bb96c12c710c24ff38b1bb9d49f637786a37845a5a45f419b22edadde78c201 Jul 29 15:47:35.173184 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.173148 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ptml9"] Jul 29 15:47:35.177480 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:35.177453 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d0e693_ccf2_4168_a63c_2f1428af95fe.slice/crio-52f06b5b915d9c16d4a5c5865da0453bc7812b8b4de70df90468beb21e7b67ee WatchSource:0}: Error finding container 52f06b5b915d9c16d4a5c5865da0453bc7812b8b4de70df90468beb21e7b67ee: Status 404 returned error can't find the container with id 52f06b5b915d9c16d4a5c5865da0453bc7812b8b4de70df90468beb21e7b67ee Jul 29 15:47:35.194391 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.194361 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gqd46"] Jul 29 15:47:35.197180 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:35.197158 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e74657_8c54_477b_a184_eb2a35c8eff5.slice/crio-d6a9d881c1eb6de3fb819888ad69a9595a44c55c5b571bd40bb7058c57c842ca WatchSource:0}: Error finding container d6a9d881c1eb6de3fb819888ad69a9595a44c55c5b571bd40bb7058c57c842ca: Status 404 returned error can't find the container with id d6a9d881c1eb6de3fb819888ad69a9595a44c55c5b571bd40bb7058c57c842ca Jul 29 15:47:35.225773 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.225739 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gqd46" event={"ID":"02e74657-8c54-477b-a184-eb2a35c8eff5","Type":"ContainerStarted","Data":"d6a9d881c1eb6de3fb819888ad69a9595a44c55c5b571bd40bb7058c57c842ca"} Jul 29 15:47:35.226861 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.226834 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29fg9" event={"ID":"69cffac7-1815-42f4-8bc7-ed8a8996b20f","Type":"ContainerStarted","Data":"2bb96c12c710c24ff38b1bb9d49f637786a37845a5a45f419b22edadde78c201"} Jul 29 15:47:35.227951 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.227923 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4n8x" event={"ID":"794e5299-4224-4873-9c41-6857ad5c2390","Type":"ContainerStarted","Data":"4a37b9929214486af29cf719b900e119ad76082838f8a39053b1d8225a460002"} Jul 29 15:47:35.228990 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:35.228954 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ptml9" event={"ID":"c6d0e693-ccf2-4168-a63c-2f1428af95fe","Type":"ContainerStarted","Data":"52f06b5b915d9c16d4a5c5865da0453bc7812b8b4de70df90468beb21e7b67ee"} Jul 29 15:47:36.195621 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.195086 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:36.195621 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.195158 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:36.200245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.200198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571cee9d-30cc-4291-91c0-abb9ef90f3c5-metrics-tls\") pod \"dns-default-xmzml\" (UID: \"571cee9d-30cc-4291-91c0-abb9ef90f3c5\") " pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:36.202081 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.202046 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"image-registry-768645b995-46b4w\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:36.236487 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.236444 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gqd46" event={"ID":"02e74657-8c54-477b-a184-eb2a35c8eff5","Type":"ContainerStarted","Data":"3b68ccc884bebd85ee1a6005dd9544f763428a6b81e8839bd338fa3c6aebf771"} Jul 29 15:47:36.236686 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.236545 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:47:36.253347 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.253291 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gqd46" podStartSLOduration=66.253271081 podStartE2EDuration="1m6.253271081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:36.252615556 +0000 UTC m=+66.793517024" watchObservedRunningTime="2026-07-29 15:47:36.253271081 +0000 UTC m=+66.794172550" Jul 29 15:47:36.420764 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.420739 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgnpm\"" Jul 29 15:47:36.427953 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.427923 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:36.433984 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.433939 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx57w\"" Jul 29 15:47:36.442394 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.442367 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:36.671988 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.671926 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-768645b995-46b4w"] Jul 29 15:47:36.675975 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:36.675943 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de6cfda_e440_4046_87eb_2f0ca0c54470.slice/crio-3574b8de2ed3a6ea4868a81adb76a75072df39c4254f5a7f34633e64430f411e WatchSource:0}: Error finding container 3574b8de2ed3a6ea4868a81adb76a75072df39c4254f5a7f34633e64430f411e: Status 404 returned error can't find the container with id 3574b8de2ed3a6ea4868a81adb76a75072df39c4254f5a7f34633e64430f411e Jul 29 15:47:36.680106 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:36.680084 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmzml"] Jul 29 15:47:36.684362 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:36.684337 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571cee9d_30cc_4291_91c0_abb9ef90f3c5.slice/crio-ca0ecd61572a63c9796231ecccc4a15b8736c4c2f9d64df698fee8656010cd5d WatchSource:0}: Error finding container ca0ecd61572a63c9796231ecccc4a15b8736c4c2f9d64df698fee8656010cd5d: Status 404 returned error can't find the container with id ca0ecd61572a63c9796231ecccc4a15b8736c4c2f9d64df698fee8656010cd5d Jul 29 15:47:37.251537 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.251456 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768645b995-46b4w" event={"ID":"4de6cfda-e440-4046-87eb-2f0ca0c54470","Type":"ContainerStarted","Data":"37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5"} Jul 29 15:47:37.251537 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.251523 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768645b995-46b4w" event={"ID":"4de6cfda-e440-4046-87eb-2f0ca0c54470","Type":"ContainerStarted","Data":"3574b8de2ed3a6ea4868a81adb76a75072df39c4254f5a7f34633e64430f411e"} Jul 29 15:47:37.252241 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.252184 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:47:37.256344 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.256305 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29fg9" event={"ID":"69cffac7-1815-42f4-8bc7-ed8a8996b20f","Type":"ContainerStarted","Data":"79cbc28248f4ccd71063ddf580387be39cdf15b9bdf1041b50ee1f7cc3f0295b"} Jul 29 15:47:37.256492 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.256374 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29fg9" event={"ID":"69cffac7-1815-42f4-8bc7-ed8a8996b20f","Type":"ContainerStarted","Data":"e4de8b0fd07e914b4ab029012223ec5c616f9f689f11a34b6fb474f9b91df584"} Jul 29 15:47:37.259641 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.259569 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmzml" event={"ID":"571cee9d-30cc-4291-91c0-abb9ef90f3c5","Type":"ContainerStarted","Data":"ca0ecd61572a63c9796231ecccc4a15b8736c4c2f9d64df698fee8656010cd5d"} Jul 29 15:47:37.278081 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.277548 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-768645b995-46b4w" podStartSLOduration=67.27752329 podStartE2EDuration="1m7.27752329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:37.275806706 +0000 UTC m=+67.816708176" watchObservedRunningTime="2026-07-29 15:47:37.27752329 +0000 UTC m=+67.818424745" Jul 29 15:47:37.295785 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:37.294233 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-29fg9" podStartSLOduration=66.040933791 podStartE2EDuration="1m7.294214138s" totalImagesPullingTime="1.253280347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:37.293191606 +0000 UTC m=+67.834093074" watchObservedRunningTime="2026-07-29 15:47:37.294214138 +0000 UTC m=+67.835115607" Jul 29 15:47:38.919018 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:38.918987 2555 scope.go:117] "RemoveContainer" containerID="776d5c3f8127ed9dd6bed028a55926cbd1ede0a7d562f316b3a39897818c39b7" Jul 29 15:47:41.864596 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:41.864547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:41.867174 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:41.867148 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabeca35-f4d0-4626-82d2-be6ded895a50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8hm8n\" (UID: \"cabeca35-f4d0-4626-82d2-be6ded895a50\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:41.965367 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:41.965272 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:41.965367 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:41.965340 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:41.966165 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:41.966140 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edc2889a-6586-468b-9fa8-f7d8ff946770-service-ca-bundle\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:41.967751 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:41.967730 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc2889a-6586-468b-9fa8-f7d8ff946770-metrics-certs\") pod \"router-default-658f857f8f-j4knh\" (UID: \"edc2889a-6586-468b-9fa8-f7d8ff946770\") " pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:42.057638 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:42.057598 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2l9p6\"" Jul 29 15:47:42.065921 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:42.065890 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" Jul 29 15:47:42.103955 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:42.103916 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dldjd\"" Jul 29 15:47:42.112138 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:42.112108 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:42.509960 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:42.509807 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n"] Jul 29 15:47:42.552036 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:42.548901 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-658f857f8f-j4knh"] Jul 29 15:47:42.554299 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:42.554243 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc2889a_6586_468b_9fa8_f7d8ff946770.slice/crio-398f514eb65592c79766af5b66afb4b450b968b9ff32e77551070797b4168167 WatchSource:0}: Error finding container 398f514eb65592c79766af5b66afb4b450b968b9ff32e77551070797b4168167: Status 404 returned error can't find the container with id 398f514eb65592c79766af5b66afb4b450b968b9ff32e77551070797b4168167 Jul 29 15:47:43.292203 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.292106 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4n8x" event={"ID":"794e5299-4224-4873-9c41-6857ad5c2390","Type":"ContainerStarted","Data":"ea47692acd576d13e3e75237c9b544798a2576dc356e20538c988b8202651e1b"} Jul 29 15:47:43.294012 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.293952 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ptml9" event={"ID":"c6d0e693-ccf2-4168-a63c-2f1428af95fe","Type":"ContainerStarted","Data":"b9548ca4b2cba3f31380775eb16e87a5acd37b76dd6b4cc0bfa740c740bad0b1"} Jul 29 15:47:43.296687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.296653 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmzml" event={"ID":"571cee9d-30cc-4291-91c0-abb9ef90f3c5","Type":"ContainerStarted","Data":"4427be93b48f82cd263074e6822f9cab4e5e426ba8211119236f6826208b5071"} Jul 29 15:47:43.296826 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.296710 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmzml" event={"ID":"571cee9d-30cc-4291-91c0-abb9ef90f3c5","Type":"ContainerStarted","Data":"7610dfeef7e2465c6c5e974bba4b126b718b4c27dba69367529f8c8a63029445"} Jul 29 15:47:43.297050 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.296996 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:43.298164 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.298138 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" event={"ID":"cabeca35-f4d0-4626-82d2-be6ded895a50","Type":"ContainerStarted","Data":"60e41bae81cd3a361a32ccd9741c92a49e4cdfe2cf363349d2ebce0433377d4d"} Jul 29 15:47:43.300931 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.300634 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-658f857f8f-j4knh" event={"ID":"edc2889a-6586-468b-9fa8-f7d8ff946770","Type":"ContainerStarted","Data":"fc59fe6779d6a709c26293e13908b00ea2898038da42a8e94f0a26b7d034762e"} Jul 29 15:47:43.300931 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.300678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-658f857f8f-j4knh" event={"ID":"edc2889a-6586-468b-9fa8-f7d8ff946770","Type":"ContainerStarted","Data":"398f514eb65592c79766af5b66afb4b450b968b9ff32e77551070797b4168167"} Jul 29 15:47:43.302812 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.302794 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 15:47:43.302926 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.302848 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" event={"ID":"d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec","Type":"ContainerStarted","Data":"01c2763665fc41b71e6b6962593c069fd1dbe9578961c564be4f0d361dc18672"} Jul 29 15:47:43.303241 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.303189 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:43.334498 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.334437 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xmzml" podStartSLOduration=33.708010492 podStartE2EDuration="39.334416514s" totalImagesPullingTime="5.626406022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:43.333721925 +0000 UTC m=+73.874623407" watchObservedRunningTime="2026-07-29 15:47:43.334416514 +0000 UTC m=+73.875317974" Jul 29 15:47:43.335804 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.335751 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ptml9" podStartSLOduration=66.177286474 podStartE2EDuration="1m13.335736343s" totalImagesPullingTime="7.158449869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:43.313212068 +0000 UTC m=+73.854113538" watchObservedRunningTime="2026-07-29 15:47:43.335736343 +0000 UTC m=+73.876637844" Jul 29 15:47:43.502214 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.502173 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-6mrvv" Jul 29 15:47:43.533222 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:43.533144 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-658f857f8f-j4knh" podStartSLOduration=34.53311882 podStartE2EDuration="34.53311882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:43.376913598 +0000 UTC m=+73.917815068" watchObservedRunningTime="2026-07-29 15:47:43.53311882 +0000 UTC m=+74.074020290" Jul 29 15:47:44.112951 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:44.112907 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:44.116637 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:44.116602 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:44.308001 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:44.307968 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:44.308953 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:44.308932 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-658f857f8f-j4knh" Jul 29 15:47:45.230637 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.230572 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r"] Jul 29 15:47:45.324320 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.324283 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r"] Jul 29 15:47:45.324780 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.324433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:45.329777 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.329746 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 15:47:45.330115 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.329771 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xkpdt\"" Jul 29 15:47:45.330279 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.329859 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 15:47:45.497139 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.497056 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb906de3-e1b3-4082-a332-b199c3fecccb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-t5f2r\" (UID: \"cb906de3-e1b3-4082-a332-b199c3fecccb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:45.497139 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.497111 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb906de3-e1b3-4082-a332-b199c3fecccb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-t5f2r\" (UID: \"cb906de3-e1b3-4082-a332-b199c3fecccb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:45.598474 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.598432 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb906de3-e1b3-4082-a332-b199c3fecccb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-t5f2r\" (UID: \"cb906de3-e1b3-4082-a332-b199c3fecccb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:45.598665 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.598522 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb906de3-e1b3-4082-a332-b199c3fecccb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-t5f2r\" (UID: \"cb906de3-e1b3-4082-a332-b199c3fecccb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:45.599220 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.599189 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb906de3-e1b3-4082-a332-b199c3fecccb-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-t5f2r\" (UID: \"cb906de3-e1b3-4082-a332-b199c3fecccb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:45.601342 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.601302 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb906de3-e1b3-4082-a332-b199c3fecccb-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-t5f2r\" (UID: \"cb906de3-e1b3-4082-a332-b199c3fecccb\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:45.638066 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:45.638032 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" Jul 29 15:47:47.270664 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:47.270634 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r"] Jul 29 15:47:47.273566 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:47.273536 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb906de3_e1b3_4082_a332_b199c3fecccb.slice/crio-71ebb7b06109021d59ef0184673ff2d5b7d06c15ac897788ffd6b5e5a5468b16 WatchSource:0}: Error finding container 71ebb7b06109021d59ef0184673ff2d5b7d06c15ac897788ffd6b5e5a5468b16: Status 404 returned error can't find the container with id 71ebb7b06109021d59ef0184673ff2d5b7d06c15ac897788ffd6b5e5a5468b16 Jul 29 15:47:47.321235 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:47.321201 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" event={"ID":"cb906de3-e1b3-4082-a332-b199c3fecccb","Type":"ContainerStarted","Data":"71ebb7b06109021d59ef0184673ff2d5b7d06c15ac897788ffd6b5e5a5468b16"} Jul 29 15:47:47.322771 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:47.322730 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" event={"ID":"cabeca35-f4d0-4626-82d2-be6ded895a50","Type":"ContainerStarted","Data":"35a8a82bcfd91044e3bd2d414df59e9abc3547b8e673392c276bd8ce6452675c"} Jul 29 15:47:47.344796 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:47.344734 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8hm8n" podStartSLOduration=33.724844934000004 podStartE2EDuration="38.344712796s" totalImagesPullingTime="4.619867862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:47.343660126 +0000 UTC m=+77.884561597" watchObservedRunningTime="2026-07-29 15:47:47.344712796 +0000 UTC m=+77.885614266" Jul 29 15:47:48.329452 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:48.329406 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4n8x" event={"ID":"794e5299-4224-4873-9c41-6857ad5c2390","Type":"ContainerStarted","Data":"ef6ef2d40f395ad564356fd1eab88573d47f7476fb685f7d61c1e3c1a0109984"} Jul 29 15:47:49.335154 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:49.335113 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" event={"ID":"cb906de3-e1b3-4082-a332-b199c3fecccb","Type":"ContainerStarted","Data":"d3f65e58bd56c5317774496f243d5515c6399d515f483e748e5a44a570d80b88"} Jul 29 15:47:49.356068 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:49.356000 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-t5f2r" podStartSLOduration=2.489635056 podStartE2EDuration="4.355977847s" totalImagesPullingTime="1.866342791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:49.354193779 +0000 UTC m=+79.895095250" watchObservedRunningTime="2026-07-29 15:47:49.355977847 +0000 UTC m=+79.896879316" Jul 29 15:47:52.349391 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:52.349353 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4n8x" event={"ID":"794e5299-4224-4873-9c41-6857ad5c2390","Type":"ContainerStarted","Data":"00f060c0733bc49c78548b5580a6f98e29c62a36a03a822d7785297dc63fef16"} Jul 29 15:47:52.369827 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:52.369745 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p4n8x" podStartSLOduration=17.853485783 podStartE2EDuration="34.369728012s" totalImagesPullingTime="16.516242229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:47:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:52.368760884 +0000 UTC m=+82.909662352" watchObservedRunningTime="2026-07-29 15:47:52.369728012 +0000 UTC m=+82.910629480" Jul 29 15:47:53.309183 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.309155 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xmzml" Jul 29 15:47:53.616997 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.616907 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f565b69cf-9n99j"] Jul 29 15:47:53.619521 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.619502 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.622393 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.622370 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 15:47:53.622538 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.622473 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 15:47:53.622669 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.622653 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 15:47:53.622737 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.622724 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 15:47:53.622798 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.622755 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 15:47:53.623056 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.623037 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 15:47:53.623851 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.623825 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 15:47:53.623983 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.623966 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7sw7x\"" Jul 29 15:47:53.630024 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.630001 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f565b69cf-9n99j"] Jul 29 15:47:53.764683 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.764638 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-oauth-serving-cert\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.764859 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.764729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-serving-cert\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.764859 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.764753 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-config\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.764859 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.764772 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-kube-api-access-xqfc4\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.764859 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.764840 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-service-ca\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.764994 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.764878 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-oauth-config\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.866156 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.866117 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-serving-cert\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.866156 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.866157 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-config\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.866363 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.866174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-kube-api-access-xqfc4\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.866363 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.866203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-service-ca\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.866461 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.866398 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-oauth-config\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.866461 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.866446 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-oauth-serving-cert\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.867056 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.866997 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-service-ca\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.867155 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.867081 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-config\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.867155 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.867134 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-oauth-serving-cert\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.868559 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.868542 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-serving-cert\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.868640 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.868624 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-oauth-config\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.879170 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.879137 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-kube-api-access-xqfc4\") pod \"console-6f565b69cf-9n99j\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:53.929015 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:53.928980 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:47:54.069547 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:54.069524 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f565b69cf-9n99j"] Jul 29 15:47:54.071489 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:47:54.071464 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbdaecc_dc55_47d5_8e03_99d6f2fda063.slice/crio-9ec1bba33f830c78f555280526be47b730be42c94f829cb60c915502b1175c4d WatchSource:0}: Error finding container 9ec1bba33f830c78f555280526be47b730be42c94f829cb60c915502b1175c4d: Status 404 returned error can't find the container with id 9ec1bba33f830c78f555280526be47b730be42c94f829cb60c915502b1175c4d Jul 29 15:47:54.358744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:54.358707 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f565b69cf-9n99j" event={"ID":"ffbdaecc-dc55-47d5-8e03-99d6f2fda063","Type":"ContainerStarted","Data":"9ec1bba33f830c78f555280526be47b730be42c94f829cb60c915502b1175c4d"} Jul 29 15:47:57.372907 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:57.372869 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f565b69cf-9n99j" event={"ID":"ffbdaecc-dc55-47d5-8e03-99d6f2fda063","Type":"ContainerStarted","Data":"b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab"} Jul 29 15:47:57.392879 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:57.392831 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f565b69cf-9n99j" podStartSLOduration=1.251925776 podStartE2EDuration="4.392811289s" totalImagesPullingTime="3.140885513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:57.391702063 +0000 UTC m=+87.932603531" watchObservedRunningTime="2026-07-29 15:47:57.392811289 +0000 UTC m=+87.933712759" Jul 29 15:47:59.275448 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:47:59.275419 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:48:02.198962 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.198920 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tshzs"] Jul 29 15:48:02.205488 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.205454 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb"] Jul 29 15:48:02.205686 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.205623 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.208247 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.208201 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 15:48:02.208612 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.208594 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 15:48:02.209207 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.209181 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 15:48:02.209304 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.209223 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.209475 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.209455 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 15:48:02.209766 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.209749 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hcxj9\"" Jul 29 15:48:02.211479 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.211452 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 15:48:02.211620 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.211555 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nl8w9\"" Jul 29 15:48:02.211687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.211631 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 15:48:02.211976 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.211961 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 15:48:02.222248 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.222220 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb"] Jul 29 15:48:02.227768 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.227739 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-root\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.227928 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.227784 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-wtmp\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.227928 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.227841 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-sys\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.227928 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.227879 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.227928 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.227906 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.228134 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.227953 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-textfile\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.228134 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.227981 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.228134 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228008 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ldcc\" (UniqueName: \"kubernetes.io/projected/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-api-access-7ldcc\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.228134 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228041 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.228134 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228076 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-metrics-client-ca\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.228134 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228131 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-tls\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.228410 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228170 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.228410 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228197 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.228410 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228274 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.228410 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.228317 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdfs\" (UniqueName: \"kubernetes.io/projected/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-kube-api-access-7pdfs\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.328976 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.328932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdfs\" (UniqueName: \"kubernetes.io/projected/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-kube-api-access-7pdfs\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.328976 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.328979 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-root\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329031 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-root\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-wtmp\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329066 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-sys\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329085 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329111 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329114 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-sys\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329134 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-textfile\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329144 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-wtmp\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329150 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.329197 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ldcc\" (UniqueName: \"kubernetes.io/projected/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-api-access-7ldcc\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:02.329222 2555 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:02.329295 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-tls podName:09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b nodeName:}" failed. No retries permitted until 2026-07-29 15:48:02.829273899 +0000 UTC m=+93.370175347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-zbwvb" (UID: "09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b") : secret "kube-state-metrics-tls" not found Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329220 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329401 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-metrics-client-ca\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329458 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-tls\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329548 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329559 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329608 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-textfile\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329644 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:02.329667 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 15:48:02.329727 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:02.329718 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-tls podName:b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9 nodeName:}" failed. No retries permitted until 2026-07-29 15:48:02.829702395 +0000 UTC m=+93.370603856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-tls") pod "node-exporter-tshzs" (UID: "b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9") : secret "node-exporter-tls" not found Jul 29 15:48:02.330316 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.329902 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.330316 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.330003 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.330316 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.330034 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-metrics-client-ca\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.330316 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.330223 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.332100 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.332074 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.332227 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.332205 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.338853 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.338819 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdfs\" (UniqueName: \"kubernetes.io/projected/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-kube-api-access-7pdfs\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.339541 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.339516 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ldcc\" (UniqueName: \"kubernetes.io/projected/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-api-access-7ldcc\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.833443 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.833385 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-tls\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:02.833661 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.833475 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.835955 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.835930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zbwvb\" (UID: \"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:02.836075 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:02.835962 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9-node-exporter-tls\") pod \"node-exporter-tshzs\" (UID: \"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9\") " pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:03.121253 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.121147 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tshzs" Jul 29 15:48:03.127309 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.127271 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" Jul 29 15:48:03.136833 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:48:03.136795 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb364fbdd_b38b_46e7_ab05_7d3e8a1b84c9.slice/crio-8d4cf6002925401bd226fcde38da9f3a7ec6096ebbd6b492f19fd55cfc44d35c WatchSource:0}: Error finding container 8d4cf6002925401bd226fcde38da9f3a7ec6096ebbd6b492f19fd55cfc44d35c: Status 404 returned error can't find the container with id 8d4cf6002925401bd226fcde38da9f3a7ec6096ebbd6b492f19fd55cfc44d35c Jul 29 15:48:03.227279 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.227240 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-648f47c7d8-mw4c4"] Jul 29 15:48:03.235018 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.234986 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.242487 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.242415 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648f47c7d8-mw4c4"] Jul 29 15:48:03.246248 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.246212 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 15:48:03.290576 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.290547 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb"] Jul 29 15:48:03.293070 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:48:03.293038 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fa9345_a3af_4cf6_a0ac_e829f6ac6d2b.slice/crio-19b489d374b82819b999e4b009a91fbd0ae83b9ed0476504101f4d3de878b5da WatchSource:0}: Error finding container 19b489d374b82819b999e4b009a91fbd0ae83b9ed0476504101f4d3de878b5da: Status 404 returned error can't find the container with id 19b489d374b82819b999e4b009a91fbd0ae83b9ed0476504101f4d3de878b5da Jul 29 15:48:03.340921 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.340885 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-oauth-config\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.341110 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.340934 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vx2l\" (UniqueName: \"kubernetes.io/projected/745f1854-ab44-438e-897c-b3634cc5b415-kube-api-access-5vx2l\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.341110 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.340976 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-trusted-ca-bundle\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.341239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.341108 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-console-config\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.341239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.341149 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-serving-cert\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.341239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.341192 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-oauth-serving-cert\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.341385 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.341247 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-service-ca\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.396551 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.396454 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tshzs" event={"ID":"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9","Type":"ContainerStarted","Data":"8d4cf6002925401bd226fcde38da9f3a7ec6096ebbd6b492f19fd55cfc44d35c"} Jul 29 15:48:03.397704 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.397671 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" event={"ID":"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b","Type":"ContainerStarted","Data":"19b489d374b82819b999e4b009a91fbd0ae83b9ed0476504101f4d3de878b5da"} Jul 29 15:48:03.442365 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.442328 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-oauth-config\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.442365 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.442363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vx2l\" (UniqueName: \"kubernetes.io/projected/745f1854-ab44-438e-897c-b3634cc5b415-kube-api-access-5vx2l\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.442677 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.442388 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-trusted-ca-bundle\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.442677 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.442490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-console-config\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.442677 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.442511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-serving-cert\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.442677 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.442544 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-oauth-serving-cert\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.442677 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.442622 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-service-ca\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.443548 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.443433 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-service-ca\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.443548 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.443437 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-console-config\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.443745 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.443686 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-oauth-serving-cert\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.444266 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.444212 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-trusted-ca-bundle\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.445371 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.445348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-oauth-config\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.446015 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.445994 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-serving-cert\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.457943 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.457909 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vx2l\" (UniqueName: \"kubernetes.io/projected/745f1854-ab44-438e-897c-b3634cc5b415-kube-api-access-5vx2l\") pod \"console-648f47c7d8-mw4c4\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.550789 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.550752 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:03.694082 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.694050 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648f47c7d8-mw4c4"] Jul 29 15:48:03.696642 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:48:03.696612 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745f1854_ab44_438e_897c_b3634cc5b415.slice/crio-406b398383a8ae3610735d826b1a2a463cb31462a5490ef8ded9b7643e5b0ada WatchSource:0}: Error finding container 406b398383a8ae3610735d826b1a2a463cb31462a5490ef8ded9b7643e5b0ada: Status 404 returned error can't find the container with id 406b398383a8ae3610735d826b1a2a463cb31462a5490ef8ded9b7643e5b0ada Jul 29 15:48:03.929363 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.929317 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:48:03.929921 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.929891 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:48:03.935733 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:03.935703 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:48:04.403757 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:04.403610 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648f47c7d8-mw4c4" event={"ID":"745f1854-ab44-438e-897c-b3634cc5b415","Type":"ContainerStarted","Data":"411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a"} Jul 29 15:48:04.403757 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:04.403671 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648f47c7d8-mw4c4" event={"ID":"745f1854-ab44-438e-897c-b3634cc5b415","Type":"ContainerStarted","Data":"406b398383a8ae3610735d826b1a2a463cb31462a5490ef8ded9b7643e5b0ada"} Jul 29 15:48:04.408499 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:04.408474 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:48:04.427452 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:04.427400 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-648f47c7d8-mw4c4" podStartSLOduration=1.427382745 podStartE2EDuration="1.427382745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:04.425507268 +0000 UTC m=+94.966408733" watchObservedRunningTime="2026-07-29 15:48:04.427382745 +0000 UTC m=+94.968284192" Jul 29 15:48:05.409991 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:05.409950 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" event={"ID":"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b","Type":"ContainerStarted","Data":"9d71ee23db8f6f94fda569717d6d016d5f09ed4569ab21cee9a339ca8a32d84b"} Jul 29 15:48:05.410494 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:05.410025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" event={"ID":"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b","Type":"ContainerStarted","Data":"3cb4e0655340bf52ae7cbbdd8ca3ffe0817f045b85eda265647bbe8ed5811f35"} Jul 29 15:48:05.410494 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:05.410055 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" event={"ID":"09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b","Type":"ContainerStarted","Data":"52d603fa571792498b87aed9803261df6a74029b6fda844c04dbba5189e48ad2"} Jul 29 15:48:05.411631 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:05.411602 2555 generic.go:356] "Generic (PLEG): container finished" podID="b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9" containerID="74e451ec1da9d314d37d7818e23f59f0c95222c783d2a4783178e22037426f0f" exitCode=0 Jul 29 15:48:05.411728 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:05.411691 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tshzs" event={"ID":"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9","Type":"ContainerDied","Data":"74e451ec1da9d314d37d7818e23f59f0c95222c783d2a4783178e22037426f0f"} Jul 29 15:48:05.436529 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:05.436485 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zbwvb" podStartSLOduration=1.7709578700000002 podStartE2EDuration="3.43646856s" totalImagesPullingTime="1.66551069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:05.435314376 +0000 UTC m=+95.976215847" watchObservedRunningTime="2026-07-29 15:48:05.43646856 +0000 UTC m=+95.977370032" Jul 29 15:48:06.418009 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:06.417965 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tshzs" event={"ID":"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9","Type":"ContainerStarted","Data":"a9d50eb706e0b7b46dc8f6e04ebef47cf556fdab6af4b0e8b4dc6b9165b48e7b"} Jul 29 15:48:06.418380 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:06.418035 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tshzs" event={"ID":"b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9","Type":"ContainerStarted","Data":"d812a88748a3497067f0256fa514a74e61387c17130e486b6bdc4661aea49097"} Jul 29 15:48:06.441300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:06.441238 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tshzs" podStartSLOduration=3.108033944 podStartE2EDuration="4.441220445s" totalImagesPullingTime="1.266801041s" totalInitContainerRuntime="66.38546ms" isStatefulPod=true podCreationTimestamp="2026-07-29 15:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:06.440863638 +0000 UTC m=+96.981765107" watchObservedRunningTime="2026-07-29 15:48:06.441220445 +0000 UTC m=+96.982121913" Jul 29 15:48:07.262660 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.262625 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gqd46" Jul 29 15:48:07.432035 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.432001 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd"] Jul 29 15:48:07.437419 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.437394 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.440870 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.440848 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 15:48:07.441097 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.441076 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 15:48:07.441152 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.441076 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7778c\"" Jul 29 15:48:07.441152 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.441078 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 15:48:07.441775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.441759 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 15:48:07.441838 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.441766 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 15:48:07.448166 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.448145 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 15:48:07.453096 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.453068 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd"] Jul 29 15:48:07.477810 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.477783 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-telemeter-client-tls\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.477992 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.477825 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-secret-telemeter-client\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.477992 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.477861 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-metrics-client-ca\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.477992 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.477938 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-serving-certs-ca-bundle\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.478147 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.478012 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.478147 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.478043 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.478147 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.478125 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-federate-client-tls\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.478298 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.478201 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxmn\" (UniqueName: \"kubernetes.io/projected/154d0538-6942-4b62-a2a0-26b36fb0d7b6-kube-api-access-jzxmn\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.579664 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.579544 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.579664 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.579620 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.579664 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.579649 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-federate-client-tls\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.579924 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.579878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxmn\" (UniqueName: \"kubernetes.io/projected/154d0538-6942-4b62-a2a0-26b36fb0d7b6-kube-api-access-jzxmn\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.579983 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.579927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-telemeter-client-tls\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.579983 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.579958 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-secret-telemeter-client\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.580081 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.580008 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-metrics-client-ca\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.580081 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.580051 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-serving-certs-ca-bundle\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.580675 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.580647 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.580809 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.580733 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-metrics-client-ca\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.580809 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.580778 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154d0538-6942-4b62-a2a0-26b36fb0d7b6-serving-certs-ca-bundle\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.582625 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.582592 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-secret-telemeter-client\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.582746 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.582729 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.582823 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.582807 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-federate-client-tls\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.582858 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.582826 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/154d0538-6942-4b62-a2a0-26b36fb0d7b6-telemeter-client-tls\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.589505 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.589476 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxmn\" (UniqueName: \"kubernetes.io/projected/154d0538-6942-4b62-a2a0-26b36fb0d7b6-kube-api-access-jzxmn\") pod \"telemeter-client-6d5bf5d8f4-tf5fd\" (UID: \"154d0538-6942-4b62-a2a0-26b36fb0d7b6\") " pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.668125 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.668089 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-768645b995-46b4w"] Jul 29 15:48:07.747339 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.747299 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" Jul 29 15:48:07.910307 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:07.910282 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd"] Jul 29 15:48:07.912202 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:48:07.912176 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154d0538_6942_4b62_a2a0_26b36fb0d7b6.slice/crio-52e5f6bbd78696933bbf0f5e6d4131786c5c72e6072563e6b8c266bcb6a7b56e WatchSource:0}: Error finding container 52e5f6bbd78696933bbf0f5e6d4131786c5c72e6072563e6b8c266bcb6a7b56e: Status 404 returned error can't find the container with id 52e5f6bbd78696933bbf0f5e6d4131786c5c72e6072563e6b8c266bcb6a7b56e Jul 29 15:48:08.427104 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:08.427065 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" event={"ID":"154d0538-6942-4b62-a2a0-26b36fb0d7b6","Type":"ContainerStarted","Data":"52e5f6bbd78696933bbf0f5e6d4131786c5c72e6072563e6b8c266bcb6a7b56e"} Jul 29 15:48:10.226434 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.226396 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648f47c7d8-mw4c4"] Jul 29 15:48:10.261663 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.261625 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f4cbbcb5f-rx9dt"] Jul 29 15:48:10.264988 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.264961 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.278419 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.278388 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f4cbbcb5f-rx9dt"] Jul 29 15:48:10.308478 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.308441 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-serving-cert\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.308478 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.308480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8rz\" (UniqueName: \"kubernetes.io/projected/b71580ec-798e-44e1-8f20-6c27bb6b55a6-kube-api-access-2x8rz\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.308730 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.308507 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-oauth-serving-cert\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.308730 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.308526 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-config\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.308730 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.308567 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-service-ca\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.308730 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.308611 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-trusted-ca-bundle\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.308730 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.308665 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-oauth-config\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.409935 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.409892 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-serving-cert\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.410118 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.409944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8rz\" (UniqueName: \"kubernetes.io/projected/b71580ec-798e-44e1-8f20-6c27bb6b55a6-kube-api-access-2x8rz\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.410118 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.409990 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-oauth-serving-cert\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.410118 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.410021 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-config\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.410118 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.410072 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-service-ca\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.410335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.410122 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-trusted-ca-bundle\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.410335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.410172 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-oauth-config\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.410902 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.410870 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-config\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.411050 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.410885 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-oauth-serving-cert\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.411050 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.410922 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-service-ca\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.411219 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.411194 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-trusted-ca-bundle\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.412834 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.412805 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-serving-cert\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.412969 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.412910 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-oauth-config\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.419175 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.419144 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8rz\" (UniqueName: \"kubernetes.io/projected/b71580ec-798e-44e1-8f20-6c27bb6b55a6-kube-api-access-2x8rz\") pod \"console-5f4cbbcb5f-rx9dt\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.439650 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.439608 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" event={"ID":"154d0538-6942-4b62-a2a0-26b36fb0d7b6","Type":"ContainerStarted","Data":"6a99ad7d3a730bab3f5225d7ae113005cfc44bfd0ba82bc6c1f4d6c899ab0243"} Jul 29 15:48:10.574151 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.574105 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:10.729617 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:10.729572 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f4cbbcb5f-rx9dt"] Jul 29 15:48:10.902638 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:48:10.902506 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71580ec_798e_44e1_8f20_6c27bb6b55a6.slice/crio-8850e3ad7c0320d07f056a0989bc01f45c7d6b1dcd461b19033fbae4d1e34449 WatchSource:0}: Error finding container 8850e3ad7c0320d07f056a0989bc01f45c7d6b1dcd461b19033fbae4d1e34449: Status 404 returned error can't find the container with id 8850e3ad7c0320d07f056a0989bc01f45c7d6b1dcd461b19033fbae4d1e34449 Jul 29 15:48:11.446012 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:11.445971 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4cbbcb5f-rx9dt" event={"ID":"b71580ec-798e-44e1-8f20-6c27bb6b55a6","Type":"ContainerStarted","Data":"956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682"} Jul 29 15:48:11.446475 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:11.446034 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4cbbcb5f-rx9dt" event={"ID":"b71580ec-798e-44e1-8f20-6c27bb6b55a6","Type":"ContainerStarted","Data":"8850e3ad7c0320d07f056a0989bc01f45c7d6b1dcd461b19033fbae4d1e34449"} Jul 29 15:48:11.448169 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:11.448145 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" event={"ID":"154d0538-6942-4b62-a2a0-26b36fb0d7b6","Type":"ContainerStarted","Data":"7658a8d3d97e655a691b172faa0e625f22230f23c75fcd6f6384a810a499e63d"} Jul 29 15:48:11.448271 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:11.448191 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" event={"ID":"154d0538-6942-4b62-a2a0-26b36fb0d7b6","Type":"ContainerStarted","Data":"e00ec515ae141fd0e85b2289dff4ef6c6dc491710af9e5651437b59395ce446d"} Jul 29 15:48:11.467308 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:11.467259 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f4cbbcb5f-rx9dt" podStartSLOduration=1.467242056 podStartE2EDuration="1.467242056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:11.466003256 +0000 UTC m=+102.006904725" watchObservedRunningTime="2026-07-29 15:48:11.467242056 +0000 UTC m=+102.008143503" Jul 29 15:48:11.490413 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:11.490366 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d5bf5d8f4-tf5fd" podStartSLOduration=1.51889769 podStartE2EDuration="4.490348706s" totalImagesPullingTime="2.971451016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:48:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:11.489462303 +0000 UTC m=+102.030363773" watchObservedRunningTime="2026-07-29 15:48:11.490348706 +0000 UTC m=+102.031250175" Jul 29 15:48:12.412183 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.412139 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f4cbbcb5f-rx9dt"] Jul 29 15:48:12.455679 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.455639 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-747fd454d7-69hgk"] Jul 29 15:48:12.525697 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.525653 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747fd454d7-69hgk"] Jul 29 15:48:12.525858 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.525757 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.632291 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.632251 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862r4\" (UniqueName: \"kubernetes.io/projected/35c7f96c-7f15-431c-b04a-724187b16e58-kube-api-access-862r4\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.632291 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.632297 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-trusted-ca-bundle\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.632504 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.632316 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-oauth-serving-cert\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.632504 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.632407 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-oauth-config\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.632504 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.632467 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-console-config\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.632636 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.632515 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-serving-cert\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.632636 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.632534 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-service-ca\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.733530 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.733446 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-console-config\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.733530 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.733487 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-serving-cert\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.733530 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.733506 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-service-ca\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.733997 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.733635 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862r4\" (UniqueName: \"kubernetes.io/projected/35c7f96c-7f15-431c-b04a-724187b16e58-kube-api-access-862r4\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.734091 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.734034 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-trusted-ca-bundle\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.734091 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.734060 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-oauth-serving-cert\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.734194 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.734098 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-oauth-config\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.734315 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.734292 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-service-ca\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.734379 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.734292 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-console-config\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.734728 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.734705 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-oauth-serving-cert\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.735046 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.735021 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-trusted-ca-bundle\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.736856 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.736834 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-serving-cert\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.736856 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.736847 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-oauth-config\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.742763 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.742740 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-862r4\" (UniqueName: \"kubernetes.io/projected/35c7f96c-7f15-431c-b04a-724187b16e58-kube-api-access-862r4\") pod \"console-747fd454d7-69hgk\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:12.834985 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:12.834953 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:13.052894 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:13.052863 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747fd454d7-69hgk"] Jul 29 15:48:13.055684 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:48:13.055658 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c7f96c_7f15_431c_b04a_724187b16e58.slice/crio-189e7fdf0b9d4bca88f3d77ea5b21c1126911f85834c848630fbc5c1c282f2a8 WatchSource:0}: Error finding container 189e7fdf0b9d4bca88f3d77ea5b21c1126911f85834c848630fbc5c1c282f2a8: Status 404 returned error can't find the container with id 189e7fdf0b9d4bca88f3d77ea5b21c1126911f85834c848630fbc5c1c282f2a8 Jul 29 15:48:13.458864 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:13.458765 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747fd454d7-69hgk" event={"ID":"35c7f96c-7f15-431c-b04a-724187b16e58","Type":"ContainerStarted","Data":"c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477"} Jul 29 15:48:13.458864 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:13.458820 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747fd454d7-69hgk" event={"ID":"35c7f96c-7f15-431c-b04a-724187b16e58","Type":"ContainerStarted","Data":"189e7fdf0b9d4bca88f3d77ea5b21c1126911f85834c848630fbc5c1c282f2a8"} Jul 29 15:48:13.487444 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:13.487390 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-747fd454d7-69hgk" podStartSLOduration=1.487370233 podStartE2EDuration="1.487370233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:13.486394961 +0000 UTC m=+104.027296441" watchObservedRunningTime="2026-07-29 15:48:13.487370233 +0000 UTC m=+104.028271703" Jul 29 15:48:13.551911 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:13.551873 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:20.574878 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:20.574837 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:22.835397 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:22.835354 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:22.835841 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:22.835488 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:22.840512 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:22.840484 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:23.502251 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:23.502157 2555 generic.go:356] "Generic (PLEG): container finished" podID="cba9d9bd-cb1c-48e0-a19d-46c76e955a4f" containerID="f0267e4c69a8c5599e1591a46f38493d332d09e22a43b32cce12485142309415" exitCode=0 Jul 29 15:48:23.502400 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:23.502240 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vrsmf" event={"ID":"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f","Type":"ContainerDied","Data":"f0267e4c69a8c5599e1591a46f38493d332d09e22a43b32cce12485142309415"} Jul 29 15:48:23.502769 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:23.502750 2555 scope.go:117] "RemoveContainer" containerID="f0267e4c69a8c5599e1591a46f38493d332d09e22a43b32cce12485142309415" Jul 29 15:48:23.506831 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:23.506807 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:48:23.573717 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:23.573681 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f565b69cf-9n99j"] Jul 29 15:48:24.430656 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:24.430624 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-658f857f8f-j4knh_edc2889a-6586-468b-9fa8-f7d8ff946770/router/0.log" Jul 29 15:48:24.444187 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:24.444161 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l7g6v_846b9e1c-7ded-4a38-86ad-61f753398cf6/serve-healthcheck-canary/0.log" Jul 29 15:48:24.508482 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:24.508444 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vrsmf" event={"ID":"cba9d9bd-cb1c-48e0-a19d-46c76e955a4f","Type":"ContainerStarted","Data":"292bf6177ec345f913bfb9b41b7c5696c9e4bf1fa640020099cffff3e07d2702"} Jul 29 15:48:32.687919 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:32.687855 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-768645b995-46b4w" podUID="4de6cfda-e440-4046-87eb-2f0ca0c54470" containerName="registry" containerID="cri-o://37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5" gracePeriod=30 Jul 29 15:48:32.934022 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:32.933996 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:48:33.018153 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018122 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-certificates\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.018357 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018175 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-trusted-ca\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.018357 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018196 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-installation-pull-secrets\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.018441 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018380 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8sn7\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-kube-api-access-q8sn7\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.018494 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018476 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-bound-sa-token\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.018540 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018518 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4de6cfda-e440-4046-87eb-2f0ca0c54470-ca-trust-extracted\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.018636 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018550 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.018636 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018577 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-image-registry-private-configuration\") pod \"4de6cfda-e440-4046-87eb-2f0ca0c54470\" (UID: \"4de6cfda-e440-4046-87eb-2f0ca0c54470\") " Jul 29 15:48:33.019055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018715 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:33.019055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018868 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:33.019055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018961 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-certificates\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.019055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.018979 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4de6cfda-e440-4046-87eb-2f0ca0c54470-trusted-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.021103 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.021068 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:33.021232 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.021131 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:48:33.021232 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.021131 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:33.021232 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.021205 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-kube-api-access-q8sn7" (OuterVolumeSpecName: "kube-api-access-q8sn7") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "kube-api-access-q8sn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:48:33.021394 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.021235 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:48:33.026306 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.026275 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de6cfda-e440-4046-87eb-2f0ca0c54470-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4de6cfda-e440-4046-87eb-2f0ca0c54470" (UID: "4de6cfda-e440-4046-87eb-2f0ca0c54470"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:48:33.120135 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.120101 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-registry-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.120135 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.120129 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-image-registry-private-configuration\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.120135 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.120139 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4de6cfda-e440-4046-87eb-2f0ca0c54470-installation-pull-secrets\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.120363 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.120148 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8sn7\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-kube-api-access-q8sn7\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.120363 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.120157 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4de6cfda-e440-4046-87eb-2f0ca0c54470-bound-sa-token\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.120363 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.120165 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4de6cfda-e440-4046-87eb-2f0ca0c54470-ca-trust-extracted\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:33.548554 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.548519 2555 generic.go:356] "Generic (PLEG): container finished" podID="4de6cfda-e440-4046-87eb-2f0ca0c54470" containerID="37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5" exitCode=0 Jul 29 15:48:33.548743 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.548559 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768645b995-46b4w" Jul 29 15:48:33.548743 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.548597 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768645b995-46b4w" event={"ID":"4de6cfda-e440-4046-87eb-2f0ca0c54470","Type":"ContainerDied","Data":"37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5"} Jul 29 15:48:33.548743 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.548642 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768645b995-46b4w" event={"ID":"4de6cfda-e440-4046-87eb-2f0ca0c54470","Type":"ContainerDied","Data":"3574b8de2ed3a6ea4868a81adb76a75072df39c4254f5a7f34633e64430f411e"} Jul 29 15:48:33.548743 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.548665 2555 scope.go:117] "RemoveContainer" containerID="37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5" Jul 29 15:48:33.557954 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.557936 2555 scope.go:117] "RemoveContainer" containerID="37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5" Jul 29 15:48:33.558224 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:33.558204 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5\": container with ID starting with 37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5 not found: ID does not exist" containerID="37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5" Jul 29 15:48:33.558293 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.558237 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5"} err="failed to get container status \"37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5\": rpc error: code = NotFound desc = could not find container \"37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5\": container with ID starting with 37953c5112d2cc855cfffc296570778c573e4d287d205d63c1b4201de2c25ea5 not found: ID does not exist" Jul 29 15:48:33.571704 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.571655 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-768645b995-46b4w"] Jul 29 15:48:33.573785 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.573761 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-768645b995-46b4w"] Jul 29 15:48:33.923546 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:33.923448 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de6cfda-e440-4046-87eb-2f0ca0c54470" path="/var/lib/kubelet/pods/4de6cfda-e440-4046-87eb-2f0ca0c54470/volumes" Jul 29 15:48:35.246365 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.246323 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-648f47c7d8-mw4c4" podUID="745f1854-ab44-438e-897c-b3634cc5b415" containerName="console" containerID="cri-o://411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a" gracePeriod=15 Jul 29 15:48:35.499484 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.499420 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:35.560769 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.560383 2555 generic.go:356] "Generic (PLEG): container finished" podID="745f1854-ab44-438e-897c-b3634cc5b415" containerID="411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a" exitCode=0 Jul 29 15:48:35.560769 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.560479 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648f47c7d8-mw4c4" event={"ID":"745f1854-ab44-438e-897c-b3634cc5b415","Type":"ContainerDied","Data":"411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a"} Jul 29 15:48:35.560769 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.560610 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648f47c7d8-mw4c4" event={"ID":"745f1854-ab44-438e-897c-b3634cc5b415","Type":"ContainerDied","Data":"406b398383a8ae3610735d826b1a2a463cb31462a5490ef8ded9b7643e5b0ada"} Jul 29 15:48:35.560769 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.560643 2555 scope.go:117] "RemoveContainer" containerID="411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a" Jul 29 15:48:35.561215 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.560976 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648f47c7d8-mw4c4" Jul 29 15:48:35.571275 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.571254 2555 scope.go:117] "RemoveContainer" containerID="411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a" Jul 29 15:48:35.571567 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:35.571540 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a\": container with ID starting with 411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a not found: ID does not exist" containerID="411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a" Jul 29 15:48:35.571706 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.571594 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a"} err="failed to get container status \"411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a\": rpc error: code = NotFound desc = could not find container \"411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a\": container with ID starting with 411b72b8a21a4f9d38425bc293d7f3ec5256bfbbcad54214e99466d7da41e61a not found: ID does not exist" Jul 29 15:48:35.648272 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648241 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vx2l\" (UniqueName: \"kubernetes.io/projected/745f1854-ab44-438e-897c-b3634cc5b415-kube-api-access-5vx2l\") pod \"745f1854-ab44-438e-897c-b3634cc5b415\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " Jul 29 15:48:35.648272 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648276 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-service-ca\") pod \"745f1854-ab44-438e-897c-b3634cc5b415\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " Jul 29 15:48:35.648515 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648301 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-oauth-serving-cert\") pod \"745f1854-ab44-438e-897c-b3634cc5b415\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " Jul 29 15:48:35.648515 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648330 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-serving-cert\") pod \"745f1854-ab44-438e-897c-b3634cc5b415\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " Jul 29 15:48:35.648515 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648401 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-console-config\") pod \"745f1854-ab44-438e-897c-b3634cc5b415\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " Jul 29 15:48:35.648515 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648444 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-oauth-config\") pod \"745f1854-ab44-438e-897c-b3634cc5b415\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " Jul 29 15:48:35.648515 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648481 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-trusted-ca-bundle\") pod \"745f1854-ab44-438e-897c-b3634cc5b415\" (UID: \"745f1854-ab44-438e-897c-b3634cc5b415\") " Jul 29 15:48:35.648846 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648818 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "745f1854-ab44-438e-897c-b3634cc5b415" (UID: "745f1854-ab44-438e-897c-b3634cc5b415"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:35.648846 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.648841 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-service-ca" (OuterVolumeSpecName: "service-ca") pod "745f1854-ab44-438e-897c-b3634cc5b415" (UID: "745f1854-ab44-438e-897c-b3634cc5b415"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:35.649130 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.649045 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-console-config" (OuterVolumeSpecName: "console-config") pod "745f1854-ab44-438e-897c-b3634cc5b415" (UID: "745f1854-ab44-438e-897c-b3634cc5b415"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:35.649259 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.649236 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "745f1854-ab44-438e-897c-b3634cc5b415" (UID: "745f1854-ab44-438e-897c-b3634cc5b415"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:35.650751 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.650714 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745f1854-ab44-438e-897c-b3634cc5b415-kube-api-access-5vx2l" (OuterVolumeSpecName: "kube-api-access-5vx2l") pod "745f1854-ab44-438e-897c-b3634cc5b415" (UID: "745f1854-ab44-438e-897c-b3634cc5b415"). InnerVolumeSpecName "kube-api-access-5vx2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:48:35.650938 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.650917 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "745f1854-ab44-438e-897c-b3634cc5b415" (UID: "745f1854-ab44-438e-897c-b3634cc5b415"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:35.651028 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.650934 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "745f1854-ab44-438e-897c-b3634cc5b415" (UID: "745f1854-ab44-438e-897c-b3634cc5b415"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:35.749604 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.749505 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:35.749604 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.749536 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-oauth-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:35.749604 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.749550 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:35.749604 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.749562 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-console-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:35.749604 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.749573 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/745f1854-ab44-438e-897c-b3634cc5b415-console-oauth-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:35.749604 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.749599 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/745f1854-ab44-438e-897c-b3634cc5b415-trusted-ca-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:35.750088 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.749612 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vx2l\" (UniqueName: \"kubernetes.io/projected/745f1854-ab44-438e-897c-b3634cc5b415-kube-api-access-5vx2l\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:35.884721 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.884689 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648f47c7d8-mw4c4"] Jul 29 15:48:35.889063 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.889036 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-648f47c7d8-mw4c4"] Jul 29 15:48:35.924034 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:35.923994 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745f1854-ab44-438e-897c-b3634cc5b415" path="/var/lib/kubelet/pods/745f1854-ab44-438e-897c-b3634cc5b415/volumes" Jul 29 15:48:38.477751 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.477712 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f4cbbcb5f-rx9dt" podUID="b71580ec-798e-44e1-8f20-6c27bb6b55a6" containerName="console" containerID="cri-o://956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682" gracePeriod=15 Jul 29 15:48:38.729769 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.729710 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:38.774068 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774037 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-oauth-serving-cert\") pod \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " Jul 29 15:48:38.774254 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774086 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-oauth-config\") pod \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " Jul 29 15:48:38.774254 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774120 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-trusted-ca-bundle\") pod \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " Jul 29 15:48:38.774254 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774154 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-config\") pod \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " Jul 29 15:48:38.774254 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774175 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-serving-cert\") pod \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " Jul 29 15:48:38.774254 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774195 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x8rz\" (UniqueName: \"kubernetes.io/projected/b71580ec-798e-44e1-8f20-6c27bb6b55a6-kube-api-access-2x8rz\") pod \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " Jul 29 15:48:38.774254 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774214 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-service-ca\") pod \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\" (UID: \"b71580ec-798e-44e1-8f20-6c27bb6b55a6\") " Jul 29 15:48:38.774636 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774605 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b71580ec-798e-44e1-8f20-6c27bb6b55a6" (UID: "b71580ec-798e-44e1-8f20-6c27bb6b55a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:38.774713 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774644 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-config" (OuterVolumeSpecName: "console-config") pod "b71580ec-798e-44e1-8f20-6c27bb6b55a6" (UID: "b71580ec-798e-44e1-8f20-6c27bb6b55a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:38.774713 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774669 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b71580ec-798e-44e1-8f20-6c27bb6b55a6" (UID: "b71580ec-798e-44e1-8f20-6c27bb6b55a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:38.774792 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.774724 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "b71580ec-798e-44e1-8f20-6c27bb6b55a6" (UID: "b71580ec-798e-44e1-8f20-6c27bb6b55a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:38.776351 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.776324 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b71580ec-798e-44e1-8f20-6c27bb6b55a6" (UID: "b71580ec-798e-44e1-8f20-6c27bb6b55a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:38.776447 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.776378 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b71580ec-798e-44e1-8f20-6c27bb6b55a6" (UID: "b71580ec-798e-44e1-8f20-6c27bb6b55a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:38.776447 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.776410 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71580ec-798e-44e1-8f20-6c27bb6b55a6-kube-api-access-2x8rz" (OuterVolumeSpecName: "kube-api-access-2x8rz") pod "b71580ec-798e-44e1-8f20-6c27bb6b55a6" (UID: "b71580ec-798e-44e1-8f20-6c27bb6b55a6"). InnerVolumeSpecName "kube-api-access-2x8rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:48:38.874851 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.874806 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-trusted-ca-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:38.874851 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.874844 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:38.874851 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.874854 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:38.875082 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.874863 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x8rz\" (UniqueName: \"kubernetes.io/projected/b71580ec-798e-44e1-8f20-6c27bb6b55a6-kube-api-access-2x8rz\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:38.875082 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.874873 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:38.875082 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.874881 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71580ec-798e-44e1-8f20-6c27bb6b55a6-oauth-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:38.875082 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:38.874889 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71580ec-798e-44e1-8f20-6c27bb6b55a6-console-oauth-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:39.582082 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.582047 2555 generic.go:356] "Generic (PLEG): container finished" podID="b71580ec-798e-44e1-8f20-6c27bb6b55a6" containerID="956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682" exitCode=0 Jul 29 15:48:39.582480 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.582093 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4cbbcb5f-rx9dt" Jul 29 15:48:39.582480 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.582133 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4cbbcb5f-rx9dt" event={"ID":"b71580ec-798e-44e1-8f20-6c27bb6b55a6","Type":"ContainerDied","Data":"956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682"} Jul 29 15:48:39.582480 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.582185 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4cbbcb5f-rx9dt" event={"ID":"b71580ec-798e-44e1-8f20-6c27bb6b55a6","Type":"ContainerDied","Data":"8850e3ad7c0320d07f056a0989bc01f45c7d6b1dcd461b19033fbae4d1e34449"} Jul 29 15:48:39.582480 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.582206 2555 scope.go:117] "RemoveContainer" containerID="956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682" Jul 29 15:48:39.591896 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.591872 2555 scope.go:117] "RemoveContainer" containerID="956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682" Jul 29 15:48:39.592262 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:39.592240 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682\": container with ID starting with 956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682 not found: ID does not exist" containerID="956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682" Jul 29 15:48:39.592324 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.592271 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682"} err="failed to get container status \"956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682\": rpc error: code = NotFound desc = could not find container \"956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682\": container with ID starting with 956af4602a5bff21d676d2f43a0acf41c39ee01b4343beda4bcea21f1e76d682 not found: ID does not exist" Jul 29 15:48:39.605998 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.605966 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f4cbbcb5f-rx9dt"] Jul 29 15:48:39.610547 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.610520 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f4cbbcb5f-rx9dt"] Jul 29 15:48:39.923988 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:39.923900 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71580ec-798e-44e1-8f20-6c27bb6b55a6" path="/var/lib/kubelet/pods/b71580ec-798e-44e1-8f20-6c27bb6b55a6/volumes" Jul 29 15:48:42.597076 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:42.597041 2555 generic.go:356] "Generic (PLEG): container finished" podID="1447efed-1b87-414e-a949-d0db6b807f42" containerID="9967afad9e8bd70076ba8ab96d042560f8b29f3bd7c8aa8d598aeb389d5895ef" exitCode=0 Jul 29 15:48:42.597467 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:42.597108 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" event={"ID":"1447efed-1b87-414e-a949-d0db6b807f42","Type":"ContainerDied","Data":"9967afad9e8bd70076ba8ab96d042560f8b29f3bd7c8aa8d598aeb389d5895ef"} Jul 29 15:48:42.597467 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:42.597427 2555 scope.go:117] "RemoveContainer" containerID="9967afad9e8bd70076ba8ab96d042560f8b29f3bd7c8aa8d598aeb389d5895ef" Jul 29 15:48:43.603209 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:43.603172 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmqwq" event={"ID":"1447efed-1b87-414e-a949-d0db6b807f42","Type":"ContainerStarted","Data":"e89e205a2b9739114dae623fae38fabc456ea0e1f296485fb3269cfba3f8fefc"} Jul 29 15:48:43.604598 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:43.604555 2555 generic.go:356] "Generic (PLEG): container finished" podID="b18913da-9faf-4eb2-bfbf-f63e9308022e" containerID="036b60279819c836d76f330138c454317acf450281372414f45cf0063b28c3c0" exitCode=0 Jul 29 15:48:43.604704 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:43.604628 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" event={"ID":"b18913da-9faf-4eb2-bfbf-f63e9308022e","Type":"ContainerDied","Data":"036b60279819c836d76f330138c454317acf450281372414f45cf0063b28c3c0"} Jul 29 15:48:43.604914 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:43.604901 2555 scope.go:117] "RemoveContainer" containerID="036b60279819c836d76f330138c454317acf450281372414f45cf0063b28c3c0" Jul 29 15:48:44.615436 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:44.615399 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp" event={"ID":"b18913da-9faf-4eb2-bfbf-f63e9308022e","Type":"ContainerStarted","Data":"c2f7b89f79d44b36e05b2985039fd9f39c844c11654aa3d22fdd7f3fab400e6f"} Jul 29 15:48:48.597229 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.597183 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f565b69cf-9n99j" podUID="ffbdaecc-dc55-47d5-8e03-99d6f2fda063" containerName="console" containerID="cri-o://b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab" gracePeriod=15 Jul 29 15:48:48.845891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.845860 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:48:48.955098 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.954999 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-service-ca\") pod \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " Jul 29 15:48:48.955098 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955069 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-oauth-serving-cert\") pod \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " Jul 29 15:48:48.955098 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955091 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-config\") pod \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " Jul 29 15:48:48.955378 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955107 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-oauth-config\") pod \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " Jul 29 15:48:48.955378 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955137 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-kube-api-access-xqfc4\") pod \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " Jul 29 15:48:48.955378 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955180 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-serving-cert\") pod \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\" (UID: \"ffbdaecc-dc55-47d5-8e03-99d6f2fda063\") " Jul 29 15:48:48.955538 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955504 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-service-ca" (OuterVolumeSpecName: "service-ca") pod "ffbdaecc-dc55-47d5-8e03-99d6f2fda063" (UID: "ffbdaecc-dc55-47d5-8e03-99d6f2fda063"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:48.955538 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955521 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ffbdaecc-dc55-47d5-8e03-99d6f2fda063" (UID: "ffbdaecc-dc55-47d5-8e03-99d6f2fda063"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:48.955538 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.955534 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-config" (OuterVolumeSpecName: "console-config") pod "ffbdaecc-dc55-47d5-8e03-99d6f2fda063" (UID: "ffbdaecc-dc55-47d5-8e03-99d6f2fda063"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:48:48.957414 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.957382 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ffbdaecc-dc55-47d5-8e03-99d6f2fda063" (UID: "ffbdaecc-dc55-47d5-8e03-99d6f2fda063"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:48.957521 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.957407 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ffbdaecc-dc55-47d5-8e03-99d6f2fda063" (UID: "ffbdaecc-dc55-47d5-8e03-99d6f2fda063"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:48:48.957521 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:48.957463 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-kube-api-access-xqfc4" (OuterVolumeSpecName: "kube-api-access-xqfc4") pod "ffbdaecc-dc55-47d5-8e03-99d6f2fda063" (UID: "ffbdaecc-dc55-47d5-8e03-99d6f2fda063"). InnerVolumeSpecName "kube-api-access-xqfc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:48:49.056198 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.056153 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-kube-api-access-xqfc4\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:49.056198 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.056192 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:49.056198 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.056203 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:49.056443 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.056212 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-oauth-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:49.056443 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.056221 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:49.056443 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.056229 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffbdaecc-dc55-47d5-8e03-99d6f2fda063-console-oauth-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:48:49.638056 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.638024 2555 generic.go:356] "Generic (PLEG): container finished" podID="ffbdaecc-dc55-47d5-8e03-99d6f2fda063" containerID="b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab" exitCode=0 Jul 29 15:48:49.638440 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.638074 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f565b69cf-9n99j" Jul 29 15:48:49.638440 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.638110 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f565b69cf-9n99j" event={"ID":"ffbdaecc-dc55-47d5-8e03-99d6f2fda063","Type":"ContainerDied","Data":"b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab"} Jul 29 15:48:49.638440 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.638161 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f565b69cf-9n99j" event={"ID":"ffbdaecc-dc55-47d5-8e03-99d6f2fda063","Type":"ContainerDied","Data":"9ec1bba33f830c78f555280526be47b730be42c94f829cb60c915502b1175c4d"} Jul 29 15:48:49.638440 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.638182 2555 scope.go:117] "RemoveContainer" containerID="b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab" Jul 29 15:48:49.647404 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.647386 2555 scope.go:117] "RemoveContainer" containerID="b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab" Jul 29 15:48:49.647766 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:48:49.647746 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab\": container with ID starting with b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab not found: ID does not exist" containerID="b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab" Jul 29 15:48:49.647828 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.647777 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab"} err="failed to get container status \"b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab\": rpc error: code = NotFound desc = could not find container \"b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab\": container with ID starting with b0da188e8c266b4d92038e3d8a3d7f0ba461969d61d00e0c928f565d3c6cb0ab not found: ID does not exist" Jul 29 15:48:49.658323 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.658294 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f565b69cf-9n99j"] Jul 29 15:48:49.662711 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.662683 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f565b69cf-9n99j"] Jul 29 15:48:49.923267 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:48:49.923177 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffbdaecc-dc55-47d5-8e03-99d6f2fda063" path="/var/lib/kubelet/pods/ffbdaecc-dc55-47d5-8e03-99d6f2fda063/volumes" Jul 29 15:49:19.747372 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747337 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ccdb6f58d-prvmk"] Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747681 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de6cfda-e440-4046-87eb-2f0ca0c54470" containerName="registry" Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747697 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de6cfda-e440-4046-87eb-2f0ca0c54470" containerName="registry" Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747714 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71580ec-798e-44e1-8f20-6c27bb6b55a6" containerName="console" Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747723 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71580ec-798e-44e1-8f20-6c27bb6b55a6" containerName="console" Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747736 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745f1854-ab44-438e-897c-b3634cc5b415" containerName="console" Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747745 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="745f1854-ab44-438e-897c-b3634cc5b415" containerName="console" Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747769 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbdaecc-dc55-47d5-8e03-99d6f2fda063" containerName="console" Jul 29 15:49:19.747850 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747779 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbdaecc-dc55-47d5-8e03-99d6f2fda063" containerName="console" Jul 29 15:49:19.748219 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747861 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbdaecc-dc55-47d5-8e03-99d6f2fda063" containerName="console" Jul 29 15:49:19.748219 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747874 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de6cfda-e440-4046-87eb-2f0ca0c54470" containerName="registry" Jul 29 15:49:19.748219 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747883 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71580ec-798e-44e1-8f20-6c27bb6b55a6" containerName="console" Jul 29 15:49:19.748219 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.747892 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="745f1854-ab44-438e-897c-b3634cc5b415" containerName="console" Jul 29 15:49:19.752627 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.752596 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:19.765023 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.764999 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccdb6f58d-prvmk"] Jul 29 15:49:19.916683 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.916630 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-serving-cert\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:19.916683 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.916683 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-oauth-config\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:19.916901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.916713 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-trusted-ca-bundle\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:19.916901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.916813 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-config\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:19.916901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.916857 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-oauth-serving-cert\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:19.916901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.916888 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt269\" (UniqueName: \"kubernetes.io/projected/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-kube-api-access-qt269\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:19.917036 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:19.916939 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-service-ca\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.017758 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.017717 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-trusted-ca-bundle\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.017935 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.017890 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-config\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.018057 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.018031 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-oauth-serving-cert\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.018203 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.018094 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt269\" (UniqueName: \"kubernetes.io/projected/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-kube-api-access-qt269\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.018203 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.018143 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-service-ca\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.018309 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.018207 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-serving-cert\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.019001 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.018962 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-config\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.019117 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.018969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-oauth-config\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.019296 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.019271 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-trusted-ca-bundle\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.019419 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.019397 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-oauth-serving-cert\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.019708 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.019684 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-service-ca\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.021863 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.021840 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-serving-cert\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.026451 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.026429 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-oauth-config\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.032845 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.032823 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt269\" (UniqueName: \"kubernetes.io/projected/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-kube-api-access-qt269\") pod \"console-ccdb6f58d-prvmk\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.064865 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.064827 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:20.198686 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.198657 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccdb6f58d-prvmk"] Jul 29 15:49:20.201250 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:49:20.201224 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d66281_8aed_45bc_a9a2_d4010f5a6a9e.slice/crio-c9c1daccacb0d13ea55b1becb25aba0722bdbc01ac4ec3ea392015f8f0b89ac7 WatchSource:0}: Error finding container c9c1daccacb0d13ea55b1becb25aba0722bdbc01ac4ec3ea392015f8f0b89ac7: Status 404 returned error can't find the container with id c9c1daccacb0d13ea55b1becb25aba0722bdbc01ac4ec3ea392015f8f0b89ac7 Jul 29 15:49:20.770763 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.770725 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb6f58d-prvmk" event={"ID":"17d66281-8aed-45bc-a9a2-d4010f5a6a9e","Type":"ContainerStarted","Data":"3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8"} Jul 29 15:49:20.771168 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.770786 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb6f58d-prvmk" event={"ID":"17d66281-8aed-45bc-a9a2-d4010f5a6a9e","Type":"ContainerStarted","Data":"c9c1daccacb0d13ea55b1becb25aba0722bdbc01ac4ec3ea392015f8f0b89ac7"} Jul 29 15:49:20.792358 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:20.792303 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ccdb6f58d-prvmk" podStartSLOduration=1.7922851309999999 podStartE2EDuration="1.792285131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:49:20.790700261 +0000 UTC m=+171.331601732" watchObservedRunningTime="2026-07-29 15:49:20.792285131 +0000 UTC m=+171.333186600" Jul 29 15:49:30.065526 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:30.065482 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:30.065912 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:30.065562 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:30.070901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:30.070873 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:30.817042 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:30.817013 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:49:30.887470 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:30.887435 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747fd454d7-69hgk"] Jul 29 15:49:55.907206 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:55.907154 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-747fd454d7-69hgk" podUID="35c7f96c-7f15-431c-b04a-724187b16e58" containerName="console" containerID="cri-o://c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477" gracePeriod=15 Jul 29 15:49:56.157898 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.157835 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:49:56.224011 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.223972 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-serving-cert\") pod \"35c7f96c-7f15-431c-b04a-724187b16e58\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " Jul 29 15:49:56.224011 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224012 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-service-ca\") pod \"35c7f96c-7f15-431c-b04a-724187b16e58\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " Jul 29 15:49:56.224234 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224055 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-862r4\" (UniqueName: \"kubernetes.io/projected/35c7f96c-7f15-431c-b04a-724187b16e58-kube-api-access-862r4\") pod \"35c7f96c-7f15-431c-b04a-724187b16e58\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " Jul 29 15:49:56.224234 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224080 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-console-config\") pod \"35c7f96c-7f15-431c-b04a-724187b16e58\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " Jul 29 15:49:56.224234 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224120 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-trusted-ca-bundle\") pod \"35c7f96c-7f15-431c-b04a-724187b16e58\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " Jul 29 15:49:56.224234 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224169 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-oauth-serving-cert\") pod \"35c7f96c-7f15-431c-b04a-724187b16e58\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " Jul 29 15:49:56.224234 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224199 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-oauth-config\") pod \"35c7f96c-7f15-431c-b04a-724187b16e58\" (UID: \"35c7f96c-7f15-431c-b04a-724187b16e58\") " Jul 29 15:49:56.224638 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224603 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-console-config" (OuterVolumeSpecName: "console-config") pod "35c7f96c-7f15-431c-b04a-724187b16e58" (UID: "35c7f96c-7f15-431c-b04a-724187b16e58"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:56.224638 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224634 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-service-ca" (OuterVolumeSpecName: "service-ca") pod "35c7f96c-7f15-431c-b04a-724187b16e58" (UID: "35c7f96c-7f15-431c-b04a-724187b16e58"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:56.224823 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224670 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "35c7f96c-7f15-431c-b04a-724187b16e58" (UID: "35c7f96c-7f15-431c-b04a-724187b16e58"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:56.224823 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.224684 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "35c7f96c-7f15-431c-b04a-724187b16e58" (UID: "35c7f96c-7f15-431c-b04a-724187b16e58"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:56.226271 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.226248 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "35c7f96c-7f15-431c-b04a-724187b16e58" (UID: "35c7f96c-7f15-431c-b04a-724187b16e58"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:56.226329 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.226277 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c7f96c-7f15-431c-b04a-724187b16e58-kube-api-access-862r4" (OuterVolumeSpecName: "kube-api-access-862r4") pod "35c7f96c-7f15-431c-b04a-724187b16e58" (UID: "35c7f96c-7f15-431c-b04a-724187b16e58"). InnerVolumeSpecName "kube-api-access-862r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:49:56.226364 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.226326 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "35c7f96c-7f15-431c-b04a-724187b16e58" (UID: "35c7f96c-7f15-431c-b04a-724187b16e58"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:56.325214 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.325175 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:49:56.325214 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.325206 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:49:56.325214 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.325216 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-862r4\" (UniqueName: \"kubernetes.io/projected/35c7f96c-7f15-431c-b04a-724187b16e58-kube-api-access-862r4\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:49:56.325214 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.325226 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-console-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:49:56.325491 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.325235 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-trusted-ca-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:49:56.325491 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.325243 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35c7f96c-7f15-431c-b04a-724187b16e58-oauth-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:49:56.325491 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.325251 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35c7f96c-7f15-431c-b04a-724187b16e58-console-oauth-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:49:56.919831 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.919797 2555 generic.go:356] "Generic (PLEG): container finished" podID="35c7f96c-7f15-431c-b04a-724187b16e58" containerID="c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477" exitCode=0 Jul 29 15:49:56.920323 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.919836 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747fd454d7-69hgk" Jul 29 15:49:56.920323 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.919840 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747fd454d7-69hgk" event={"ID":"35c7f96c-7f15-431c-b04a-724187b16e58","Type":"ContainerDied","Data":"c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477"} Jul 29 15:49:56.920323 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.919957 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747fd454d7-69hgk" event={"ID":"35c7f96c-7f15-431c-b04a-724187b16e58","Type":"ContainerDied","Data":"189e7fdf0b9d4bca88f3d77ea5b21c1126911f85834c848630fbc5c1c282f2a8"} Jul 29 15:49:56.920323 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.919983 2555 scope.go:117] "RemoveContainer" containerID="c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477" Jul 29 15:49:56.929053 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.929030 2555 scope.go:117] "RemoveContainer" containerID="c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477" Jul 29 15:49:56.929346 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:49:56.929323 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477\": container with ID starting with c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477 not found: ID does not exist" containerID="c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477" Jul 29 15:49:56.929417 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.929360 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477"} err="failed to get container status \"c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477\": rpc error: code = NotFound desc = could not find container \"c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477\": container with ID starting with c11536a3c571314d4501e7662ef917c610a355dc9ac2371f4019f3970f877477 not found: ID does not exist" Jul 29 15:49:56.942499 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.942441 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747fd454d7-69hgk"] Jul 29 15:49:56.944324 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:56.944299 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-747fd454d7-69hgk"] Jul 29 15:49:57.924124 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:49:57.924090 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c7f96c-7f15-431c-b04a-724187b16e58" path="/var/lib/kubelet/pods/35c7f96c-7f15-431c-b04a-724187b16e58/volumes" Jul 29 15:50:27.409635 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.409597 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s"] Jul 29 15:50:27.410115 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.409926 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c7f96c-7f15-431c-b04a-724187b16e58" containerName="console" Jul 29 15:50:27.410115 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.409937 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c7f96c-7f15-431c-b04a-724187b16e58" containerName="console" Jul 29 15:50:27.410115 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.409984 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c7f96c-7f15-431c-b04a-724187b16e58" containerName="console" Jul 29 15:50:27.412930 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.412907 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.415673 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.415650 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 15:50:27.416813 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.416794 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9vjkx\"" Jul 29 15:50:27.416957 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.416822 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 15:50:27.421609 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.421567 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s"] Jul 29 15:50:27.477233 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.477195 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.477406 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.477243 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.477406 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.477306 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964m9\" (UniqueName: \"kubernetes.io/projected/8126f918-e4be-409b-bef3-e3b2d82cd062-kube-api-access-964m9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.577687 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.577643 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.577876 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.577705 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.577876 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.577748 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964m9\" (UniqueName: \"kubernetes.io/projected/8126f918-e4be-409b-bef3-e3b2d82cd062-kube-api-access-964m9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.578117 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.578093 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.578160 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.578104 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.586322 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.586295 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-964m9\" (UniqueName: \"kubernetes.io/projected/8126f918-e4be-409b-bef3-e3b2d82cd062-kube-api-access-964m9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.723451 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.723363 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:27.859885 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:27.859852 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s"] Jul 29 15:50:27.863261 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:50:27.863230 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8126f918_e4be_409b_bef3_e3b2d82cd062.slice/crio-f5e8e4a6b668c297c8ffa0ed2c7653368708dd2be1bf343439eed8f915a9aeea WatchSource:0}: Error finding container f5e8e4a6b668c297c8ffa0ed2c7653368708dd2be1bf343439eed8f915a9aeea: Status 404 returned error can't find the container with id f5e8e4a6b668c297c8ffa0ed2c7653368708dd2be1bf343439eed8f915a9aeea Jul 29 15:50:28.042772 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:28.042729 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" event={"ID":"8126f918-e4be-409b-bef3-e3b2d82cd062","Type":"ContainerStarted","Data":"f5e8e4a6b668c297c8ffa0ed2c7653368708dd2be1bf343439eed8f915a9aeea"} Jul 29 15:50:35.073985 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:35.073943 2555 generic.go:356] "Generic (PLEG): container finished" podID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerID="db28b29aa5adf960cae07cb2b938d500b8b11e6b4ad084867be083955c9dbcb1" exitCode=0 Jul 29 15:50:35.074372 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:35.074025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" event={"ID":"8126f918-e4be-409b-bef3-e3b2d82cd062","Type":"ContainerDied","Data":"db28b29aa5adf960cae07cb2b938d500b8b11e6b4ad084867be083955c9dbcb1"} Jul 29 15:50:38.087513 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:38.087475 2555 generic.go:356] "Generic (PLEG): container finished" podID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerID="f41eabcbe0e783082b4b9a4862ac0c1a89ca9dbd760e0a2bf60df4c6c65cbc8b" exitCode=0 Jul 29 15:50:38.088015 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:38.087563 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" event={"ID":"8126f918-e4be-409b-bef3-e3b2d82cd062","Type":"ContainerDied","Data":"f41eabcbe0e783082b4b9a4862ac0c1a89ca9dbd760e0a2bf60df4c6c65cbc8b"} Jul 29 15:50:45.121761 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:45.121726 2555 generic.go:356] "Generic (PLEG): container finished" podID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerID="39d7d6f19bc6b8b1905b875f3fe0d9c19e4b61c9698344964391c2a8828a9e0f" exitCode=0 Jul 29 15:50:45.121761 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:45.121763 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" event={"ID":"8126f918-e4be-409b-bef3-e3b2d82cd062","Type":"ContainerDied","Data":"39d7d6f19bc6b8b1905b875f3fe0d9c19e4b61c9698344964391c2a8828a9e0f"} Jul 29 15:50:46.250959 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.250934 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:46.338044 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.338008 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-util\") pod \"8126f918-e4be-409b-bef3-e3b2d82cd062\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " Jul 29 15:50:46.338212 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.338085 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-bundle\") pod \"8126f918-e4be-409b-bef3-e3b2d82cd062\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " Jul 29 15:50:46.338212 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.338118 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-964m9\" (UniqueName: \"kubernetes.io/projected/8126f918-e4be-409b-bef3-e3b2d82cd062-kube-api-access-964m9\") pod \"8126f918-e4be-409b-bef3-e3b2d82cd062\" (UID: \"8126f918-e4be-409b-bef3-e3b2d82cd062\") " Jul 29 15:50:46.338784 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.338750 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-bundle" (OuterVolumeSpecName: "bundle") pod "8126f918-e4be-409b-bef3-e3b2d82cd062" (UID: "8126f918-e4be-409b-bef3-e3b2d82cd062"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:50:46.340338 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.340306 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8126f918-e4be-409b-bef3-e3b2d82cd062-kube-api-access-964m9" (OuterVolumeSpecName: "kube-api-access-964m9") pod "8126f918-e4be-409b-bef3-e3b2d82cd062" (UID: "8126f918-e4be-409b-bef3-e3b2d82cd062"). InnerVolumeSpecName "kube-api-access-964m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:50:46.343774 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.343739 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-util" (OuterVolumeSpecName: "util") pod "8126f918-e4be-409b-bef3-e3b2d82cd062" (UID: "8126f918-e4be-409b-bef3-e3b2d82cd062"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:50:46.439091 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.439007 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:50:46.439091 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.439034 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964m9\" (UniqueName: \"kubernetes.io/projected/8126f918-e4be-409b-bef3-e3b2d82cd062-kube-api-access-964m9\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:50:46.439091 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:46.439043 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8126f918-e4be-409b-bef3-e3b2d82cd062-util\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:50:47.131296 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:47.131267 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" Jul 29 15:50:47.131466 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:47.131256 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vm47s" event={"ID":"8126f918-e4be-409b-bef3-e3b2d82cd062","Type":"ContainerDied","Data":"f5e8e4a6b668c297c8ffa0ed2c7653368708dd2be1bf343439eed8f915a9aeea"} Jul 29 15:50:47.131466 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:47.131376 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e8e4a6b668c297c8ffa0ed2c7653368708dd2be1bf343439eed8f915a9aeea" Jul 29 15:50:54.269789 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.269708 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-thlfx"] Jul 29 15:50:54.270245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.269990 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerName="extract" Jul 29 15:50:54.270245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.270000 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerName="extract" Jul 29 15:50:54.270245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.270012 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerName="util" Jul 29 15:50:54.270245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.270020 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerName="util" Jul 29 15:50:54.270245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.270031 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerName="pull" Jul 29 15:50:54.270245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.270037 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerName="pull" Jul 29 15:50:54.270245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.270105 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8126f918-e4be-409b-bef3-e3b2d82cd062" containerName="extract" Jul 29 15:50:54.292471 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.292435 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-thlfx"] Jul 29 15:50:54.292635 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.292531 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.296879 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.296853 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 15:50:54.297757 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.297741 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 15:50:54.297836 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.297745 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-f7j6p\"" Jul 29 15:50:54.399502 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.399462 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvgh\" (UniqueName: \"kubernetes.io/projected/61e50546-f0d7-444a-9d66-6155f2733faf-kube-api-access-cpvgh\") pod \"cert-manager-webhook-6486ffd997-thlfx\" (UID: \"61e50546-f0d7-444a-9d66-6155f2733faf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.399708 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.399518 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61e50546-f0d7-444a-9d66-6155f2733faf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-thlfx\" (UID: \"61e50546-f0d7-444a-9d66-6155f2733faf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.500456 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.500418 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvgh\" (UniqueName: \"kubernetes.io/projected/61e50546-f0d7-444a-9d66-6155f2733faf-kube-api-access-cpvgh\") pod \"cert-manager-webhook-6486ffd997-thlfx\" (UID: \"61e50546-f0d7-444a-9d66-6155f2733faf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.500630 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.500477 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61e50546-f0d7-444a-9d66-6155f2733faf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-thlfx\" (UID: \"61e50546-f0d7-444a-9d66-6155f2733faf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.521190 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.521114 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvgh\" (UniqueName: \"kubernetes.io/projected/61e50546-f0d7-444a-9d66-6155f2733faf-kube-api-access-cpvgh\") pod \"cert-manager-webhook-6486ffd997-thlfx\" (UID: \"61e50546-f0d7-444a-9d66-6155f2733faf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.526801 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.526777 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61e50546-f0d7-444a-9d66-6155f2733faf-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-thlfx\" (UID: \"61e50546-f0d7-444a-9d66-6155f2733faf\") " pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.610282 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.610243 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:54.779534 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:50:54.779504 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61e50546_f0d7_444a_9d66_6155f2733faf.slice/crio-b0514135c1194e2eedc15bb6b006bc0f60066df08e27b28596af140b76812113 WatchSource:0}: Error finding container b0514135c1194e2eedc15bb6b006bc0f60066df08e27b28596af140b76812113: Status 404 returned error can't find the container with id b0514135c1194e2eedc15bb6b006bc0f60066df08e27b28596af140b76812113 Jul 29 15:50:54.779766 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:54.779747 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-thlfx"] Jul 29 15:50:55.163823 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:55.163733 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" event={"ID":"61e50546-f0d7-444a-9d66-6155f2733faf","Type":"ContainerStarted","Data":"b0514135c1194e2eedc15bb6b006bc0f60066df08e27b28596af140b76812113"} Jul 29 15:50:58.178098 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:58.177997 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" event={"ID":"61e50546-f0d7-444a-9d66-6155f2733faf","Type":"ContainerStarted","Data":"864f5ecb71a2455fce56736d804abe472375aa84cadd63c22e26ca61734b987c"} Jul 29 15:50:58.178098 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:58.178065 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:50:58.195416 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:50:58.195362 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" podStartSLOduration=1.098210724 podStartE2EDuration="4.19534723s" totalImagesPullingTime="3.097136506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:50:58.194714937 +0000 UTC m=+268.735616408" watchObservedRunningTime="2026-07-29 15:50:58.19534723 +0000 UTC m=+268.736248699" Jul 29 15:51:04.183659 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:04.183625 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-thlfx" Jul 29 15:51:14.668173 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.668137 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8bcmz"] Jul 29 15:51:14.670483 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.670462 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:14.673495 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.673465 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7dxt5\"" Jul 29 15:51:14.682355 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.682324 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8bcmz"] Jul 29 15:51:14.777164 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.777118 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qxx\" (UniqueName: \"kubernetes.io/projected/78e5cf5c-9c8e-450e-a022-9fbad736cd6c-kube-api-access-p6qxx\") pod \"cert-manager-55b4d954d5-8bcmz\" (UID: \"78e5cf5c-9c8e-450e-a022-9fbad736cd6c\") " pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:14.777344 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.777172 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78e5cf5c-9c8e-450e-a022-9fbad736cd6c-bound-sa-token\") pod \"cert-manager-55b4d954d5-8bcmz\" (UID: \"78e5cf5c-9c8e-450e-a022-9fbad736cd6c\") " pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:14.877750 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.877712 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qxx\" (UniqueName: \"kubernetes.io/projected/78e5cf5c-9c8e-450e-a022-9fbad736cd6c-kube-api-access-p6qxx\") pod \"cert-manager-55b4d954d5-8bcmz\" (UID: \"78e5cf5c-9c8e-450e-a022-9fbad736cd6c\") " pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:14.877750 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.877754 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78e5cf5c-9c8e-450e-a022-9fbad736cd6c-bound-sa-token\") pod \"cert-manager-55b4d954d5-8bcmz\" (UID: \"78e5cf5c-9c8e-450e-a022-9fbad736cd6c\") " pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:14.886270 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.886234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78e5cf5c-9c8e-450e-a022-9fbad736cd6c-bound-sa-token\") pod \"cert-manager-55b4d954d5-8bcmz\" (UID: \"78e5cf5c-9c8e-450e-a022-9fbad736cd6c\") " pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:14.886394 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.886305 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qxx\" (UniqueName: \"kubernetes.io/projected/78e5cf5c-9c8e-450e-a022-9fbad736cd6c-kube-api-access-p6qxx\") pod \"cert-manager-55b4d954d5-8bcmz\" (UID: \"78e5cf5c-9c8e-450e-a022-9fbad736cd6c\") " pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:14.978917 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:14.978818 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8bcmz" Jul 29 15:51:15.110574 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:15.110503 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8bcmz"] Jul 29 15:51:15.114270 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:51:15.114237 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78e5cf5c_9c8e_450e_a022_9fbad736cd6c.slice/crio-8cf0fb943cd64c4834242d743a8f524dddb370b140967803d8ec54f52719bb81 WatchSource:0}: Error finding container 8cf0fb943cd64c4834242d743a8f524dddb370b140967803d8ec54f52719bb81: Status 404 returned error can't find the container with id 8cf0fb943cd64c4834242d743a8f524dddb370b140967803d8ec54f52719bb81 Jul 29 15:51:15.243290 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:15.243200 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8bcmz" event={"ID":"78e5cf5c-9c8e-450e-a022-9fbad736cd6c","Type":"ContainerStarted","Data":"65c6ccf5a3a6e61be4215f1ceeabe6b06002fb98939d2438d1e4728d6f8026dc"} Jul 29 15:51:15.243290 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:15.243243 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8bcmz" event={"ID":"78e5cf5c-9c8e-450e-a022-9fbad736cd6c","Type":"ContainerStarted","Data":"8cf0fb943cd64c4834242d743a8f524dddb370b140967803d8ec54f52719bb81"} Jul 29 15:51:15.260507 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:15.260426 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-8bcmz" podStartSLOduration=1.2604030179999999 podStartE2EDuration="1.260403018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:51:15.260258539 +0000 UTC m=+285.801160008" watchObservedRunningTime="2026-07-29 15:51:15.260403018 +0000 UTC m=+285.801304491" Jul 29 15:51:29.830553 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:29.830523 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 15:51:29.834631 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:29.834604 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 15:51:29.843170 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:29.843146 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 15:51:39.740987 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.740953 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4"] Jul 29 15:51:39.743337 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.743321 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.745974 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.745951 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9vjkx\"" Jul 29 15:51:39.746080 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.745987 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 15:51:39.747095 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.747078 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 15:51:39.753455 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.753434 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4"] Jul 29 15:51:39.779335 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.779296 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.779514 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.779352 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/745bb684-3dde-4ff3-9522-f7edaa0d7af6-kube-api-access-4tbkw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.779514 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.779442 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.880288 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.880250 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/745bb684-3dde-4ff3-9522-f7edaa0d7af6-kube-api-access-4tbkw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.880439 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.880305 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.880439 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.880335 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.880689 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.880673 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.880726 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.880713 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:39.888674 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:39.888642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/745bb684-3dde-4ff3-9522-f7edaa0d7af6-kube-api-access-4tbkw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:40.053427 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:40.053391 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:40.187645 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:40.187614 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4"] Jul 29 15:51:40.190126 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:51:40.190097 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745bb684_3dde_4ff3_9522_f7edaa0d7af6.slice/crio-ff147275bc4e62d94677ebfbc1cb48fa69f93c3ca540af28ba6044fc7a524fa1 WatchSource:0}: Error finding container ff147275bc4e62d94677ebfbc1cb48fa69f93c3ca540af28ba6044fc7a524fa1: Status 404 returned error can't find the container with id ff147275bc4e62d94677ebfbc1cb48fa69f93c3ca540af28ba6044fc7a524fa1 Jul 29 15:51:40.341860 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:40.341772 2555 generic.go:356] "Generic (PLEG): container finished" podID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerID="618b1944ff2b35364d3ba6fd28f9f29a2435c56421ec0d5e363e3068cd0993e9" exitCode=0 Jul 29 15:51:40.341860 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:40.341793 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" event={"ID":"745bb684-3dde-4ff3-9522-f7edaa0d7af6","Type":"ContainerDied","Data":"618b1944ff2b35364d3ba6fd28f9f29a2435c56421ec0d5e363e3068cd0993e9"} Jul 29 15:51:40.341860 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:40.341855 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" event={"ID":"745bb684-3dde-4ff3-9522-f7edaa0d7af6","Type":"ContainerStarted","Data":"ff147275bc4e62d94677ebfbc1cb48fa69f93c3ca540af28ba6044fc7a524fa1"} Jul 29 15:51:40.342300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:40.342286 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 15:51:41.347931 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:41.347893 2555 generic.go:356] "Generic (PLEG): container finished" podID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerID="f14c8bf46771eb95f0e6c5301fbd4cb263b7801ac5af5cd400c06fcdc3f1a9cb" exitCode=0 Jul 29 15:51:41.348326 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:41.347969 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" event={"ID":"745bb684-3dde-4ff3-9522-f7edaa0d7af6","Type":"ContainerDied","Data":"f14c8bf46771eb95f0e6c5301fbd4cb263b7801ac5af5cd400c06fcdc3f1a9cb"} Jul 29 15:51:42.354049 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:42.354010 2555 generic.go:356] "Generic (PLEG): container finished" podID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerID="e55487300a3df8800694bef14f2aee644eacb9dca19bc3f6470e0e0693d59ecb" exitCode=0 Jul 29 15:51:42.354447 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:42.354055 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" event={"ID":"745bb684-3dde-4ff3-9522-f7edaa0d7af6","Type":"ContainerDied","Data":"e55487300a3df8800694bef14f2aee644eacb9dca19bc3f6470e0e0693d59ecb"} Jul 29 15:51:43.484991 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.484967 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:43.508168 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.508136 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/745bb684-3dde-4ff3-9522-f7edaa0d7af6-kube-api-access-4tbkw\") pod \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " Jul 29 15:51:43.508325 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.508227 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-util\") pod \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " Jul 29 15:51:43.508325 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.508288 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-bundle\") pod \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\" (UID: \"745bb684-3dde-4ff3-9522-f7edaa0d7af6\") " Jul 29 15:51:43.509372 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.509336 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-bundle" (OuterVolumeSpecName: "bundle") pod "745bb684-3dde-4ff3-9522-f7edaa0d7af6" (UID: "745bb684-3dde-4ff3-9522-f7edaa0d7af6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:51:43.511800 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.511751 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745bb684-3dde-4ff3-9522-f7edaa0d7af6-kube-api-access-4tbkw" (OuterVolumeSpecName: "kube-api-access-4tbkw") pod "745bb684-3dde-4ff3-9522-f7edaa0d7af6" (UID: "745bb684-3dde-4ff3-9522-f7edaa0d7af6"). InnerVolumeSpecName "kube-api-access-4tbkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:51:43.515014 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.514980 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-util" (OuterVolumeSpecName: "util") pod "745bb684-3dde-4ff3-9522-f7edaa0d7af6" (UID: "745bb684-3dde-4ff3-9522-f7edaa0d7af6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:51:43.610402 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.610314 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-util\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:51:43.610402 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.610344 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/745bb684-3dde-4ff3-9522-f7edaa0d7af6-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:51:43.610402 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:43.610352 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/745bb684-3dde-4ff3-9522-f7edaa0d7af6-kube-api-access-4tbkw\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:51:44.365473 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.365434 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" event={"ID":"745bb684-3dde-4ff3-9522-f7edaa0d7af6","Type":"ContainerDied","Data":"ff147275bc4e62d94677ebfbc1cb48fa69f93c3ca540af28ba6044fc7a524fa1"} Jul 29 15:51:44.365633 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.365478 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5s7s4" Jul 29 15:51:44.365633 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.365488 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff147275bc4e62d94677ebfbc1cb48fa69f93c3ca540af28ba6044fc7a524fa1" Jul 29 15:51:44.704953 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.704874 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp"] Jul 29 15:51:44.705300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.705168 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerName="pull" Jul 29 15:51:44.705300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.705179 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerName="pull" Jul 29 15:51:44.705300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.705195 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerName="util" Jul 29 15:51:44.705300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.705200 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerName="util" Jul 29 15:51:44.705300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.705210 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerName="extract" Jul 29 15:51:44.705300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.705216 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerName="extract" Jul 29 15:51:44.705300 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.705261 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="745bb684-3dde-4ff3-9522-f7edaa0d7af6" containerName="extract" Jul 29 15:51:44.707430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.707408 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.710470 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.710444 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8psf9\"" Jul 29 15:51:44.710912 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.710893 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 15:51:44.711973 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.711955 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 15:51:44.712066 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.711973 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:51:44.712161 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.712148 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 15:51:44.712871 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.712852 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 15:51:44.719010 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.718990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr6m\" (UniqueName: \"kubernetes.io/projected/959ab688-2096-4d1a-a44f-aa18d790afe4-kube-api-access-plr6m\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.719104 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.719020 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/959ab688-2096-4d1a-a44f-aa18d790afe4-cert\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.719104 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.719039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/959ab688-2096-4d1a-a44f-aa18d790afe4-manager-config\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.719217 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.719100 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/959ab688-2096-4d1a-a44f-aa18d790afe4-metrics-cert\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.729168 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.729138 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp"] Jul 29 15:51:44.819713 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.819668 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/959ab688-2096-4d1a-a44f-aa18d790afe4-metrics-cert\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.819919 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.819786 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr6m\" (UniqueName: \"kubernetes.io/projected/959ab688-2096-4d1a-a44f-aa18d790afe4-kube-api-access-plr6m\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.819919 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.819823 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/959ab688-2096-4d1a-a44f-aa18d790afe4-cert\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.819919 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.819847 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/959ab688-2096-4d1a-a44f-aa18d790afe4-manager-config\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.820475 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.820449 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/959ab688-2096-4d1a-a44f-aa18d790afe4-manager-config\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.822234 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.822212 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/959ab688-2096-4d1a-a44f-aa18d790afe4-cert\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.822330 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.822244 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/959ab688-2096-4d1a-a44f-aa18d790afe4-metrics-cert\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:44.830264 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:44.830233 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr6m\" (UniqueName: \"kubernetes.io/projected/959ab688-2096-4d1a-a44f-aa18d790afe4-kube-api-access-plr6m\") pod \"lws-controller-manager-7cb94b984b-q7qjp\" (UID: \"959ab688-2096-4d1a-a44f-aa18d790afe4\") " pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:45.017959 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:45.017922 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:45.200966 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:45.200932 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp"] Jul 29 15:51:45.204098 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:51:45.204071 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959ab688_2096_4d1a_a44f_aa18d790afe4.slice/crio-ef7fea1ba0194dd13b9e021877dd271769a1638aeac66181b56bd36e8b6505d1 WatchSource:0}: Error finding container ef7fea1ba0194dd13b9e021877dd271769a1638aeac66181b56bd36e8b6505d1: Status 404 returned error can't find the container with id ef7fea1ba0194dd13b9e021877dd271769a1638aeac66181b56bd36e8b6505d1 Jul 29 15:51:45.371095 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:45.371012 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" event={"ID":"959ab688-2096-4d1a-a44f-aa18d790afe4","Type":"ContainerStarted","Data":"ef7fea1ba0194dd13b9e021877dd271769a1638aeac66181b56bd36e8b6505d1"} Jul 29 15:51:48.388745 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:48.388707 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" event={"ID":"959ab688-2096-4d1a-a44f-aa18d790afe4","Type":"ContainerStarted","Data":"19608440d3f0400b91c92b5b4ef7509eee55c2350f66c40efe07b7af9dcdf583"} Jul 29 15:51:48.389165 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:48.388850 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:48.431916 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:48.431863 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" podStartSLOduration=2.069240758 podStartE2EDuration="4.431843224s" totalImagesPullingTime="2.362602466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:51:48.431056929 +0000 UTC m=+318.971958398" watchObservedRunningTime="2026-07-29 15:51:48.431843224 +0000 UTC m=+318.972744697" Jul 29 15:51:54.971182 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:54.971148 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b"] Jul 29 15:51:54.973554 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:54.973537 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:54.979163 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:54.979139 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 15:51:54.979281 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:54.979185 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 15:51:54.979281 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:54.979217 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 15:51:54.979675 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:54.979650 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k69gf\"" Jul 29 15:51:54.979794 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:54.979674 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 15:51:55.006387 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.006337 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d95a3de-1277-4bb4-9668-606f736ca6c1-tmp\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.006563 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.006446 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d95a3de-1277-4bb4-9668-606f736ca6c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.006563 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.006524 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d95a3de-1277-4bb4-9668-606f736ca6c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.006563 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.006553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2cl\" (UniqueName: \"kubernetes.io/projected/8d95a3de-1277-4bb4-9668-606f736ca6c1-kube-api-access-sm2cl\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.006762 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.006631 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d95a3de-1277-4bb4-9668-606f736ca6c1-manifests\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.008732 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.008704 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b"] Jul 29 15:51:55.107031 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.106988 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d95a3de-1277-4bb4-9668-606f736ca6c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.107031 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.107028 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2cl\" (UniqueName: \"kubernetes.io/projected/8d95a3de-1277-4bb4-9668-606f736ca6c1-kube-api-access-sm2cl\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.107248 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.107064 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d95a3de-1277-4bb4-9668-606f736ca6c1-manifests\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.107248 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.107089 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d95a3de-1277-4bb4-9668-606f736ca6c1-tmp\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.107248 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.107112 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d95a3de-1277-4bb4-9668-606f736ca6c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.107562 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.107533 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d95a3de-1277-4bb4-9668-606f736ca6c1-manifests\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.109537 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.109505 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d95a3de-1277-4bb4-9668-606f736ca6c1-tmp\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.109649 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.109640 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d95a3de-1277-4bb4-9668-606f736ca6c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.109744 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.109727 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d95a3de-1277-4bb4-9668-606f736ca6c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.122826 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.122798 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2cl\" (UniqueName: \"kubernetes.io/projected/8d95a3de-1277-4bb4-9668-606f736ca6c1-kube-api-access-sm2cl\") pod \"opendatahub-operator-controller-manager-75b794959f-mkm2b\" (UID: \"8d95a3de-1277-4bb4-9668-606f736ca6c1\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.283419 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.283382 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:51:55.453948 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:55.453923 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b"] Jul 29 15:51:55.456664 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:51:55.456637 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d95a3de_1277_4bb4_9668_606f736ca6c1.slice/crio-62b470f6ec161dca0f9ea7cc038c3162f0f4bd19c3a8d20b08e545ae00d275b2 WatchSource:0}: Error finding container 62b470f6ec161dca0f9ea7cc038c3162f0f4bd19c3a8d20b08e545ae00d275b2: Status 404 returned error can't find the container with id 62b470f6ec161dca0f9ea7cc038c3162f0f4bd19c3a8d20b08e545ae00d275b2 Jul 29 15:51:56.426992 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:56.426952 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" event={"ID":"8d95a3de-1277-4bb4-9668-606f736ca6c1","Type":"ContainerStarted","Data":"62b470f6ec161dca0f9ea7cc038c3162f0f4bd19c3a8d20b08e545ae00d275b2"} Jul 29 15:51:59.396520 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:59.396485 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cb94b984b-q7qjp" Jul 29 15:51:59.443296 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:51:59.443262 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" event={"ID":"8d95a3de-1277-4bb4-9668-606f736ca6c1","Type":"ContainerStarted","Data":"50c537d999ffc6921f2167f40bd67baaec838282cdb5a7b132848bacc56b0da7"} Jul 29 15:52:00.450136 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:00.450101 2555 generic.go:356] "Generic (PLEG): container finished" podID="8d95a3de-1277-4bb4-9668-606f736ca6c1" containerID="50c537d999ffc6921f2167f40bd67baaec838282cdb5a7b132848bacc56b0da7" exitCode=0 Jul 29 15:52:00.450566 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:00.450191 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" event={"ID":"8d95a3de-1277-4bb4-9668-606f736ca6c1","Type":"ContainerDied","Data":"50c537d999ffc6921f2167f40bd67baaec838282cdb5a7b132848bacc56b0da7"} Jul 29 15:52:01.457153 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:01.457120 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" event={"ID":"8d95a3de-1277-4bb4-9668-606f736ca6c1","Type":"ContainerStarted","Data":"dd103b4b14f62ea5d99a8244f3c76cc16554d4f02171406e5dcddd87f0332613"} Jul 29 15:52:01.457565 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:01.457235 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:52:01.480407 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:01.480350 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" podStartSLOduration=3.093304206 podStartE2EDuration="7.480327473s" totalImagesPullingTime="3.315412325s" totalInitContainerRuntime="1.071610942s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:51:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:01.478789413 +0000 UTC m=+332.019690882" watchObservedRunningTime="2026-07-29 15:52:01.480327473 +0000 UTC m=+332.021228943" Jul 29 15:52:12.464176 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:12.464142 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-mkm2b" Jul 29 15:52:23.886706 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.886623 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67"] Jul 29 15:52:23.889113 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.889086 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:23.918086 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.918055 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 15:52:23.918577 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.918551 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 15:52:23.918749 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.918721 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 15:52:23.919369 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.919351 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 15:52:23.924053 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.924027 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-clx7j\"" Jul 29 15:52:23.929529 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:23.926162 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67"] Jul 29 15:52:24.061152 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.061106 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.061152 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.061150 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.061399 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.061245 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.061399 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.061304 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxsp\" (UniqueName: \"kubernetes.io/projected/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-kube-api-access-fwxsp\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.061517 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.061392 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.061517 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.061435 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.061517 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.061486 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.162829 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.162728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.162829 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.162780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxsp\" (UniqueName: \"kubernetes.io/projected/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-kube-api-access-fwxsp\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.162829 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.162799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.162829 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.162817 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.163113 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.162849 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.163113 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.162879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.163113 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.162896 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.163570 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.163542 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.165203 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.165170 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.165203 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.165181 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.165530 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.165514 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.165686 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.165662 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.188688 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.188656 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxsp\" (UniqueName: \"kubernetes.io/projected/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-kube-api-access-fwxsp\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.191768 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.191749 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hlz67\" (UID: \"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.198709 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.198685 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:24.382364 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.382325 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67"] Jul 29 15:52:24.383728 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:52:24.383701 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca8c4b0_cc3e_4516_b4a5_1111fdfbb11a.slice/crio-8fd72b131e568a8c60a2890a66c94492990fa1fe41a105cf3473e65c67498dc2 WatchSource:0}: Error finding container 8fd72b131e568a8c60a2890a66c94492990fa1fe41a105cf3473e65c67498dc2: Status 404 returned error can't find the container with id 8fd72b131e568a8c60a2890a66c94492990fa1fe41a105cf3473e65c67498dc2 Jul 29 15:52:24.557648 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:24.557612 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" event={"ID":"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a","Type":"ContainerStarted","Data":"8fd72b131e568a8c60a2890a66c94492990fa1fe41a105cf3473e65c67498dc2"} Jul 29 15:52:26.852949 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:26.852904 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 29 15:52:26.853255 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:26.852986 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 29 15:52:27.574603 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:27.574550 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" event={"ID":"8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a","Type":"ContainerStarted","Data":"f526766fb5bb94b0b67de4a5a06ed563e187fa14a73a44450cdaa69c625aeddb"} Jul 29 15:52:27.574814 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:27.574707 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:52:27.576461 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:27.576431 2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-hlz67 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 15:52:27.576619 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:27.576515 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" podUID="8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 15:52:27.639439 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:27.639377 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" podStartSLOduration=2.172957645 podStartE2EDuration="4.639360495s" totalImagesPullingTime="2.46640285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:27.638097658 +0000 UTC m=+358.178999130" watchObservedRunningTime="2026-07-29 15:52:27.639360495 +0000 UTC m=+358.180261965" Jul 29 15:52:28.580415 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:52:28.580388 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hlz67" Jul 29 15:53:31.738881 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.738845 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2"] Jul 29 15:53:31.742271 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.742255 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.744873 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.744852 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 15:53:31.745946 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.745925 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 15:53:31.746057 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.745946 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-52j2g\"" Jul 29 15:53:31.751469 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.751446 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2"] Jul 29 15:53:31.856852 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.856809 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6cgz\" (UniqueName: \"kubernetes.io/projected/e9c3688c-416b-4570-b346-7c7f8713a69a-kube-api-access-z6cgz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.856852 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.856852 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.857111 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.856955 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.958192 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.958150 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6cgz\" (UniqueName: \"kubernetes.io/projected/e9c3688c-416b-4570-b346-7c7f8713a69a-kube-api-access-z6cgz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.958192 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.958193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.958446 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.958259 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.958666 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.958645 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.958666 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.958659 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:31.967181 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:31.967158 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6cgz\" (UniqueName: \"kubernetes.io/projected/e9c3688c-416b-4570-b346-7c7f8713a69a-kube-api-access-z6cgz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:32.051549 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.051507 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:32.130893 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.130865 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd"] Jul 29 15:53:32.139385 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.139352 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.144897 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.144830 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd"] Jul 29 15:53:32.196356 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.196331 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2"] Jul 29 15:53:32.198233 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:53:32.198205 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c3688c_416b_4570_b346_7c7f8713a69a.slice/crio-3ba968fe89c6406bc2558d4a4c38236b3371cd14972557b1531b08d4dc758b9f WatchSource:0}: Error finding container 3ba968fe89c6406bc2558d4a4c38236b3371cd14972557b1531b08d4dc758b9f: Status 404 returned error can't find the container with id 3ba968fe89c6406bc2558d4a4c38236b3371cd14972557b1531b08d4dc758b9f Jul 29 15:53:32.260368 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.260339 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.260487 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.260383 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.260487 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.260423 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-472jg\" (UniqueName: \"kubernetes.io/projected/916f64ed-b951-416d-8f90-4d2243c2c7be-kube-api-access-472jg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.361874 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.361832 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.362070 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.361887 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.362070 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.361949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472jg\" (UniqueName: \"kubernetes.io/projected/916f64ed-b951-416d-8f90-4d2243c2c7be-kube-api-access-472jg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.362302 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.362281 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.362375 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.362309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.376344 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.376319 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-472jg\" (UniqueName: \"kubernetes.io/projected/916f64ed-b951-416d-8f90-4d2243c2c7be-kube-api-access-472jg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.453441 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.453404 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:32.590343 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.590314 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd"] Jul 29 15:53:32.593006 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:53:32.592971 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916f64ed_b951_416d_8f90_4d2243c2c7be.slice/crio-1513705e8c9358119565ff6eca813c47aff8f71cf090fe66c2c2e6258ab7ba15 WatchSource:0}: Error finding container 1513705e8c9358119565ff6eca813c47aff8f71cf090fe66c2c2e6258ab7ba15: Status 404 returned error can't find the container with id 1513705e8c9358119565ff6eca813c47aff8f71cf090fe66c2c2e6258ab7ba15 Jul 29 15:53:32.595925 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.595793 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5"] Jul 29 15:53:32.600884 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.600860 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.614633 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.614607 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5"] Jul 29 15:53:32.663741 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.663702 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.663901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.663762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjbq\" (UniqueName: \"kubernetes.io/projected/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-kube-api-access-pqjbq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.663901 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.663885 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.765118 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.765081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.765623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.765138 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.765623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.765180 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjbq\" (UniqueName: \"kubernetes.io/projected/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-kube-api-access-pqjbq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.765623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.765519 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.765766 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.765622 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.774361 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.774330 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjbq\" (UniqueName: \"kubernetes.io/projected/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-kube-api-access-pqjbq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:32.870996 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.870900 2555 generic.go:356] "Generic (PLEG): container finished" podID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerID="bb6bfcc9702750d7c76d45c7acca1eca166d8387eca09e65c2670fc117a6e641" exitCode=0 Jul 29 15:53:32.871161 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.870997 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" event={"ID":"916f64ed-b951-416d-8f90-4d2243c2c7be","Type":"ContainerDied","Data":"bb6bfcc9702750d7c76d45c7acca1eca166d8387eca09e65c2670fc117a6e641"} Jul 29 15:53:32.871161 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.871048 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" event={"ID":"916f64ed-b951-416d-8f90-4d2243c2c7be","Type":"ContainerStarted","Data":"1513705e8c9358119565ff6eca813c47aff8f71cf090fe66c2c2e6258ab7ba15"} Jul 29 15:53:32.872632 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.872609 2555 generic.go:356] "Generic (PLEG): container finished" podID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerID="db39c23cbe3c9006e5a1c272d6763b883ec34c9beb1bc6584c89e96470a547ea" exitCode=0 Jul 29 15:53:32.872737 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.872652 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" event={"ID":"e9c3688c-416b-4570-b346-7c7f8713a69a","Type":"ContainerDied","Data":"db39c23cbe3c9006e5a1c272d6763b883ec34c9beb1bc6584c89e96470a547ea"} Jul 29 15:53:32.872737 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.872696 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" event={"ID":"e9c3688c-416b-4570-b346-7c7f8713a69a","Type":"ContainerStarted","Data":"3ba968fe89c6406bc2558d4a4c38236b3371cd14972557b1531b08d4dc758b9f"} Jul 29 15:53:32.912756 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:32.912722 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:33.051323 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:33.051297 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5"] Jul 29 15:53:33.052764 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:53:33.052738 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb23bd3_186e_4ba1_b8ae_359f9afd4d21.slice/crio-95ec1530b94ca52c9dae93f492ee8f9a3fd3c5dba858c05a77dc5bf089e37dfb WatchSource:0}: Error finding container 95ec1530b94ca52c9dae93f492ee8f9a3fd3c5dba858c05a77dc5bf089e37dfb: Status 404 returned error can't find the container with id 95ec1530b94ca52c9dae93f492ee8f9a3fd3c5dba858c05a77dc5bf089e37dfb Jul 29 15:53:33.880346 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:33.880321 2555 generic.go:356] "Generic (PLEG): container finished" podID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerID="6e97fb88afe13c920a5a9edd4ef8f2f77812e983443b7a88c0aa2dbc2a4ac9a4" exitCode=0 Jul 29 15:53:33.880695 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:33.880357 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" event={"ID":"916f64ed-b951-416d-8f90-4d2243c2c7be","Type":"ContainerDied","Data":"6e97fb88afe13c920a5a9edd4ef8f2f77812e983443b7a88c0aa2dbc2a4ac9a4"} Jul 29 15:53:33.882107 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:33.882086 2555 generic.go:356] "Generic (PLEG): container finished" podID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerID="5760cfe38c3ab7cc5155268518910e483d397af44888d264e67814c31484847a" exitCode=0 Jul 29 15:53:33.882195 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:33.882117 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" event={"ID":"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21","Type":"ContainerDied","Data":"5760cfe38c3ab7cc5155268518910e483d397af44888d264e67814c31484847a"} Jul 29 15:53:33.882195 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:33.882146 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" event={"ID":"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21","Type":"ContainerStarted","Data":"95ec1530b94ca52c9dae93f492ee8f9a3fd3c5dba858c05a77dc5bf089e37dfb"} Jul 29 15:53:34.889625 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:34.889521 2555 generic.go:356] "Generic (PLEG): container finished" podID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerID="90ff40810edcf0112c44cf03f8212806df289bfa20b93fcd5b4cc33abad6c3c5" exitCode=0 Jul 29 15:53:34.889625 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:34.889565 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" event={"ID":"e9c3688c-416b-4570-b346-7c7f8713a69a","Type":"ContainerDied","Data":"90ff40810edcf0112c44cf03f8212806df289bfa20b93fcd5b4cc33abad6c3c5"} Jul 29 15:53:34.891868 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:34.891845 2555 generic.go:356] "Generic (PLEG): container finished" podID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerID="bd82c1557151cff87b122a778b13af57828953a0e5977128f9b17b621a88fec3" exitCode=0 Jul 29 15:53:34.891957 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:34.891929 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" event={"ID":"916f64ed-b951-416d-8f90-4d2243c2c7be","Type":"ContainerDied","Data":"bd82c1557151cff87b122a778b13af57828953a0e5977128f9b17b621a88fec3"} Jul 29 15:53:34.893690 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:34.893669 2555 generic.go:356] "Generic (PLEG): container finished" podID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerID="b4effc23ae2acc6739dcfe01603c7e2c15979e0626c9342e4317956b85d09478" exitCode=0 Jul 29 15:53:34.893783 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:34.893714 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" event={"ID":"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21","Type":"ContainerDied","Data":"b4effc23ae2acc6739dcfe01603c7e2c15979e0626c9342e4317956b85d09478"} Jul 29 15:53:35.900553 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:35.900521 2555 generic.go:356] "Generic (PLEG): container finished" podID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerID="84479761f05c4c1bd5fd8d1e929d7b9b778c050187484b1b181b5a4531c27152" exitCode=0 Jul 29 15:53:35.900997 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:35.900611 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" event={"ID":"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21","Type":"ContainerDied","Data":"84479761f05c4c1bd5fd8d1e929d7b9b778c050187484b1b181b5a4531c27152"} Jul 29 15:53:35.902407 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:35.902385 2555 generic.go:356] "Generic (PLEG): container finished" podID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerID="05243bfaf9e599cc351d555d830f358c182e87bbb46ffd61e0b6ff269e44ae6e" exitCode=0 Jul 29 15:53:35.902519 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:35.902437 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" event={"ID":"e9c3688c-416b-4570-b346-7c7f8713a69a","Type":"ContainerDied","Data":"05243bfaf9e599cc351d555d830f358c182e87bbb46ffd61e0b6ff269e44ae6e"} Jul 29 15:53:36.033551 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.033524 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:36.095843 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.095806 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-util\") pod \"916f64ed-b951-416d-8f90-4d2243c2c7be\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " Jul 29 15:53:36.096055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.095879 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-472jg\" (UniqueName: \"kubernetes.io/projected/916f64ed-b951-416d-8f90-4d2243c2c7be-kube-api-access-472jg\") pod \"916f64ed-b951-416d-8f90-4d2243c2c7be\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " Jul 29 15:53:36.096055 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.095914 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-bundle\") pod \"916f64ed-b951-416d-8f90-4d2243c2c7be\" (UID: \"916f64ed-b951-416d-8f90-4d2243c2c7be\") " Jul 29 15:53:36.096561 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.096532 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-bundle" (OuterVolumeSpecName: "bundle") pod "916f64ed-b951-416d-8f90-4d2243c2c7be" (UID: "916f64ed-b951-416d-8f90-4d2243c2c7be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:53:36.098168 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.098145 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916f64ed-b951-416d-8f90-4d2243c2c7be-kube-api-access-472jg" (OuterVolumeSpecName: "kube-api-access-472jg") pod "916f64ed-b951-416d-8f90-4d2243c2c7be" (UID: "916f64ed-b951-416d-8f90-4d2243c2c7be"). InnerVolumeSpecName "kube-api-access-472jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:53:36.100947 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.100919 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-util" (OuterVolumeSpecName: "util") pod "916f64ed-b951-416d-8f90-4d2243c2c7be" (UID: "916f64ed-b951-416d-8f90-4d2243c2c7be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:53:36.196852 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.196758 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-util\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:36.196852 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.196792 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-472jg\" (UniqueName: \"kubernetes.io/projected/916f64ed-b951-416d-8f90-4d2243c2c7be-kube-api-access-472jg\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:36.196852 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.196805 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/916f64ed-b951-416d-8f90-4d2243c2c7be-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:36.909130 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.909099 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" Jul 29 15:53:36.909130 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.909120 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd" event={"ID":"916f64ed-b951-416d-8f90-4d2243c2c7be","Type":"ContainerDied","Data":"1513705e8c9358119565ff6eca813c47aff8f71cf090fe66c2c2e6258ab7ba15"} Jul 29 15:53:36.909666 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:36.909165 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1513705e8c9358119565ff6eca813c47aff8f71cf090fe66c2c2e6258ab7ba15" Jul 29 15:53:37.073440 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.073337 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:37.075355 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.075335 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:37.205669 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.205535 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-util\") pod \"e9c3688c-416b-4570-b346-7c7f8713a69a\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " Jul 29 15:53:37.205669 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.205636 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-bundle\") pod \"e9c3688c-416b-4570-b346-7c7f8713a69a\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " Jul 29 15:53:37.205900 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.205687 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqjbq\" (UniqueName: \"kubernetes.io/projected/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-kube-api-access-pqjbq\") pod \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " Jul 29 15:53:37.205900 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.205712 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-util\") pod \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " Jul 29 15:53:37.205900 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.205760 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-bundle\") pod \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\" (UID: \"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21\") " Jul 29 15:53:37.205900 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.205796 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6cgz\" (UniqueName: \"kubernetes.io/projected/e9c3688c-416b-4570-b346-7c7f8713a69a-kube-api-access-z6cgz\") pod \"e9c3688c-416b-4570-b346-7c7f8713a69a\" (UID: \"e9c3688c-416b-4570-b346-7c7f8713a69a\") " Jul 29 15:53:37.206124 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.206093 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-bundle" (OuterVolumeSpecName: "bundle") pod "e9c3688c-416b-4570-b346-7c7f8713a69a" (UID: "e9c3688c-416b-4570-b346-7c7f8713a69a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:53:37.206354 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.206324 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-bundle" (OuterVolumeSpecName: "bundle") pod "cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" (UID: "cdb23bd3-186e-4ba1-b8ae-359f9afd4d21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:53:37.207935 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.207909 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c3688c-416b-4570-b346-7c7f8713a69a-kube-api-access-z6cgz" (OuterVolumeSpecName: "kube-api-access-z6cgz") pod "e9c3688c-416b-4570-b346-7c7f8713a69a" (UID: "e9c3688c-416b-4570-b346-7c7f8713a69a"). InnerVolumeSpecName "kube-api-access-z6cgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:53:37.208223 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.208204 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-kube-api-access-pqjbq" (OuterVolumeSpecName: "kube-api-access-pqjbq") pod "cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" (UID: "cdb23bd3-186e-4ba1-b8ae-359f9afd4d21"). InnerVolumeSpecName "kube-api-access-pqjbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:53:37.211812 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.211780 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-util" (OuterVolumeSpecName: "util") pod "e9c3688c-416b-4570-b346-7c7f8713a69a" (UID: "e9c3688c-416b-4570-b346-7c7f8713a69a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:53:37.211913 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.211879 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-util" (OuterVolumeSpecName: "util") pod "cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" (UID: "cdb23bd3-186e-4ba1-b8ae-359f9afd4d21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:53:37.307211 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.307155 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:37.307211 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.307198 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6cgz\" (UniqueName: \"kubernetes.io/projected/e9c3688c-416b-4570-b346-7c7f8713a69a-kube-api-access-z6cgz\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:37.307211 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.307210 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-util\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:37.307211 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.307218 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9c3688c-416b-4570-b346-7c7f8713a69a-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:37.307211 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.307227 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjbq\" (UniqueName: \"kubernetes.io/projected/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-kube-api-access-pqjbq\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:37.307512 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.307237 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cdb23bd3-186e-4ba1-b8ae-359f9afd4d21-util\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:53:37.915691 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.915664 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" Jul 29 15:53:37.915691 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.915677 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5" event={"ID":"cdb23bd3-186e-4ba1-b8ae-359f9afd4d21","Type":"ContainerDied","Data":"95ec1530b94ca52c9dae93f492ee8f9a3fd3c5dba858c05a77dc5bf089e37dfb"} Jul 29 15:53:37.916220 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.915724 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95ec1530b94ca52c9dae93f492ee8f9a3fd3c5dba858c05a77dc5bf089e37dfb" Jul 29 15:53:37.917557 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.917534 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" Jul 29 15:53:37.917709 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.917534 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2" event={"ID":"e9c3688c-416b-4570-b346-7c7f8713a69a","Type":"ContainerDied","Data":"3ba968fe89c6406bc2558d4a4c38236b3371cd14972557b1531b08d4dc758b9f"} Jul 29 15:53:37.917709 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:53:37.917629 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ba968fe89c6406bc2558d4a4c38236b3371cd14972557b1531b08d4dc758b9f" Jul 29 15:54:18.181906 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.181874 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb"] Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182181 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerName="util" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182192 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerName="util" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182200 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerName="pull" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182205 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerName="pull" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182219 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerName="util" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182224 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerName="util" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182231 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerName="pull" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182236 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerName="pull" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182245 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerName="extract" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182250 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerName="extract" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182258 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerName="extract" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182263 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerName="extract" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182268 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerName="extract" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182274 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerName="extract" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182282 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerName="util" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182287 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerName="util" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182292 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerName="pull" Jul 29 15:54:18.182328 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182296 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerName="pull" Jul 29 15:54:18.182925 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182345 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9c3688c-416b-4570-b346-7c7f8713a69a" containerName="extract" Jul 29 15:54:18.182925 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182354 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb23bd3-186e-4ba1-b8ae-359f9afd4d21" containerName="extract" Jul 29 15:54:18.182925 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.182361 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="916f64ed-b951-416d-8f90-4d2243c2c7be" containerName="extract" Jul 29 15:54:18.186887 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.186870 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.192345 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.192323 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 15:54:18.192662 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.192641 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 15:54:18.193272 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.193084 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j69w2\"" Jul 29 15:54:18.199442 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.199412 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb"] Jul 29 15:54:18.265711 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.265669 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgdf\" (UniqueName: \"kubernetes.io/projected/7feca122-2f3c-45df-9ac2-95414fcfe5ee-kube-api-access-9rgdf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.265889 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.265730 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7feca122-2f3c-45df-9ac2-95414fcfe5ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.366788 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.366754 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgdf\" (UniqueName: \"kubernetes.io/projected/7feca122-2f3c-45df-9ac2-95414fcfe5ee-kube-api-access-9rgdf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.366954 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.366804 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7feca122-2f3c-45df-9ac2-95414fcfe5ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.367147 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.367131 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7feca122-2f3c-45df-9ac2-95414fcfe5ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.375430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.375400 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgdf\" (UniqueName: \"kubernetes.io/projected/7feca122-2f3c-45df-9ac2-95414fcfe5ee-kube-api-access-9rgdf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.497167 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.497083 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:18.636116 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.636088 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb"] Jul 29 15:54:18.638370 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:54:18.638343 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7feca122_2f3c_45df_9ac2_95414fcfe5ee.slice/crio-e28c68bfaf26b8172d32e65cdc767196477d4c38f7e9781ed9f301a4cf453704 WatchSource:0}: Error finding container e28c68bfaf26b8172d32e65cdc767196477d4c38f7e9781ed9f301a4cf453704: Status 404 returned error can't find the container with id e28c68bfaf26b8172d32e65cdc767196477d4c38f7e9781ed9f301a4cf453704 Jul 29 15:54:18.820990 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.820957 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb"] Jul 29 15:54:18.824365 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.824338 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb"] Jul 29 15:54:18.849451 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.849415 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt"] Jul 29 15:54:18.854391 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.854364 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:18.867712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.867682 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt"] Jul 29 15:54:18.972775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.972744 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7280f82-126b-4f8f-b2a1-934a40d46273-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lwrtt\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:18.972955 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:18.972813 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fz7\" (UniqueName: \"kubernetes.io/projected/e7280f82-126b-4f8f-b2a1-934a40d46273-kube-api-access-w8fz7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lwrtt\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:19.073412 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:19.073323 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7280f82-126b-4f8f-b2a1-934a40d46273-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lwrtt\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:19.073627 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:19.073425 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fz7\" (UniqueName: \"kubernetes.io/projected/e7280f82-126b-4f8f-b2a1-934a40d46273-kube-api-access-w8fz7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lwrtt\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:19.073749 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:19.073729 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7280f82-126b-4f8f-b2a1-934a40d46273-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lwrtt\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:19.083256 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:19.083224 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fz7\" (UniqueName: \"kubernetes.io/projected/e7280f82-126b-4f8f-b2a1-934a40d46273-kube-api-access-w8fz7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lwrtt\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:19.167049 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:19.166999 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:19.312274 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:19.312250 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt"] Jul 29 15:54:19.315963 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:54:19.315935 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7280f82_126b_4f8f_b2a1_934a40d46273.slice/crio-537703f5d790a41a798e55a529c1bb2757b2606e021af614cc4d05b2a0c7dfd9 WatchSource:0}: Error finding container 537703f5d790a41a798e55a529c1bb2757b2606e021af614cc4d05b2a0c7dfd9: Status 404 returned error can't find the container with id 537703f5d790a41a798e55a529c1bb2757b2606e021af614cc4d05b2a0c7dfd9 Jul 29 15:54:20.123724 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:20.123682 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" event={"ID":"e7280f82-126b-4f8f-b2a1-934a40d46273","Type":"ContainerStarted","Data":"537703f5d790a41a798e55a529c1bb2757b2606e021af614cc4d05b2a0c7dfd9"} Jul 29 15:54:23.146085 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.145981 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" podUID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" containerName="manager" containerID="cri-o://bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8" gracePeriod=2 Jul 29 15:54:23.147712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.147684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" event={"ID":"e7280f82-126b-4f8f-b2a1-934a40d46273","Type":"ContainerStarted","Data":"5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89"} Jul 29 15:54:23.147842 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.147824 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:23.172059 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.172006 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" podStartSLOduration=1.623135774 podStartE2EDuration="5.171989702s" totalImagesPullingTime="3.548853928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:54:23.169234365 +0000 UTC m=+473.710135834" watchObservedRunningTime="2026-07-29 15:54:23.171989702 +0000 UTC m=+473.712891171" Jul 29 15:54:23.386441 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.386411 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:23.389016 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:54:23.388990 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:23.513656 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.513627 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rgdf\" (UniqueName: \"kubernetes.io/projected/7feca122-2f3c-45df-9ac2-95414fcfe5ee-kube-api-access-9rgdf\") pod \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " Jul 29 15:54:23.513836 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.513731 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7feca122-2f3c-45df-9ac2-95414fcfe5ee-extensions-socket-volume\") pod \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\" (UID: \"7feca122-2f3c-45df-9ac2-95414fcfe5ee\") " Jul 29 15:54:23.514028 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.513998 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7feca122-2f3c-45df-9ac2-95414fcfe5ee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7feca122-2f3c-45df-9ac2-95414fcfe5ee" (UID: "7feca122-2f3c-45df-9ac2-95414fcfe5ee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:54:23.515762 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.515737 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feca122-2f3c-45df-9ac2-95414fcfe5ee-kube-api-access-9rgdf" (OuterVolumeSpecName: "kube-api-access-9rgdf") pod "7feca122-2f3c-45df-9ac2-95414fcfe5ee" (UID: "7feca122-2f3c-45df-9ac2-95414fcfe5ee"). InnerVolumeSpecName "kube-api-access-9rgdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:54:23.614724 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.614679 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7feca122-2f3c-45df-9ac2-95414fcfe5ee-extensions-socket-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:54:23.614724 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.614717 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgdf\" (UniqueName: \"kubernetes.io/projected/7feca122-2f3c-45df-9ac2-95414fcfe5ee-kube-api-access-9rgdf\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:54:23.924146 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:23.924111 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" path="/var/lib/kubelet/pods/7feca122-2f3c-45df-9ac2-95414fcfe5ee/volumes" Jul 29 15:54:24.153683 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:24.153594 2555 generic.go:356] "Generic (PLEG): container finished" podID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" containerID="bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8" exitCode=2 Jul 29 15:54:24.153683 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:24.153651 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:24.154158 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:24.153694 2555 scope.go:117] "RemoveContainer" containerID="bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8" Jul 29 15:54:24.160745 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:54:24.160715 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-fwbbb\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fwbbb" Jul 29 15:54:24.162255 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:24.162243 2555 scope.go:117] "RemoveContainer" containerID="bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8" Jul 29 15:54:24.162560 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:54:24.162539 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8\": container with ID starting with bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8 not found: ID does not exist" containerID="bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8" Jul 29 15:54:24.162617 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:24.162569 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8"} err="failed to get container status \"bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8\": rpc error: code = NotFound desc = could not find container \"bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8\": container with ID starting with bb630b8db6c05cf1d9f9c40458799483abc078fdb0bab078b4f3a47b88fbf7c8 not found: ID does not exist" Jul 29 15:54:34.156961 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:34.156929 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:46.383245 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.383205 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt"] Jul 29 15:54:46.383841 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.383436 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" podUID="e7280f82-126b-4f8f-b2a1-934a40d46273" containerName="manager" containerID="cri-o://5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89" gracePeriod=10 Jul 29 15:54:46.639143 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.639083 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:46.824737 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.824702 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8fz7\" (UniqueName: \"kubernetes.io/projected/e7280f82-126b-4f8f-b2a1-934a40d46273-kube-api-access-w8fz7\") pod \"e7280f82-126b-4f8f-b2a1-934a40d46273\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " Jul 29 15:54:46.824910 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.824825 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7280f82-126b-4f8f-b2a1-934a40d46273-extensions-socket-volume\") pod \"e7280f82-126b-4f8f-b2a1-934a40d46273\" (UID: \"e7280f82-126b-4f8f-b2a1-934a40d46273\") " Jul 29 15:54:46.825184 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.825161 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7280f82-126b-4f8f-b2a1-934a40d46273-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7280f82-126b-4f8f-b2a1-934a40d46273" (UID: "e7280f82-126b-4f8f-b2a1-934a40d46273"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:54:46.826881 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.826858 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7280f82-126b-4f8f-b2a1-934a40d46273-kube-api-access-w8fz7" (OuterVolumeSpecName: "kube-api-access-w8fz7") pod "e7280f82-126b-4f8f-b2a1-934a40d46273" (UID: "e7280f82-126b-4f8f-b2a1-934a40d46273"). InnerVolumeSpecName "kube-api-access-w8fz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:54:46.926408 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.926318 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7280f82-126b-4f8f-b2a1-934a40d46273-extensions-socket-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:54:46.926408 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:46.926349 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fz7\" (UniqueName: \"kubernetes.io/projected/e7280f82-126b-4f8f-b2a1-934a40d46273-kube-api-access-w8fz7\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:54:47.269594 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.269551 2555 generic.go:356] "Generic (PLEG): container finished" podID="e7280f82-126b-4f8f-b2a1-934a40d46273" containerID="5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89" exitCode=0 Jul 29 15:54:47.269758 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.269613 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" Jul 29 15:54:47.269758 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.269619 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" event={"ID":"e7280f82-126b-4f8f-b2a1-934a40d46273","Type":"ContainerDied","Data":"5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89"} Jul 29 15:54:47.269758 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.269673 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt" event={"ID":"e7280f82-126b-4f8f-b2a1-934a40d46273","Type":"ContainerDied","Data":"537703f5d790a41a798e55a529c1bb2757b2606e021af614cc4d05b2a0c7dfd9"} Jul 29 15:54:47.269758 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.269696 2555 scope.go:117] "RemoveContainer" containerID="5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89" Jul 29 15:54:47.279303 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.279283 2555 scope.go:117] "RemoveContainer" containerID="5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89" Jul 29 15:54:47.279617 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:54:47.279577 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89\": container with ID starting with 5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89 not found: ID does not exist" containerID="5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89" Jul 29 15:54:47.279662 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.279626 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89"} err="failed to get container status \"5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89\": rpc error: code = NotFound desc = could not find container \"5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89\": container with ID starting with 5ad76ac71ff1f54fc09427abcc1637b281fa33d16ecac5fe25f95d9c54c13f89 not found: ID does not exist" Jul 29 15:54:47.294178 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.294145 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt"] Jul 29 15:54:47.304558 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.304526 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lwrtt"] Jul 29 15:54:47.924064 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:54:47.924036 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7280f82-126b-4f8f-b2a1-934a40d46273" path="/var/lib/kubelet/pods/e7280f82-126b-4f8f-b2a1-934a40d46273/volumes" Jul 29 15:55:02.502633 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.502596 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpngp"] Jul 29 15:55:02.503022 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.502914 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7280f82-126b-4f8f-b2a1-934a40d46273" containerName="manager" Jul 29 15:55:02.503022 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.502924 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7280f82-126b-4f8f-b2a1-934a40d46273" containerName="manager" Jul 29 15:55:02.503022 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.502938 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" containerName="manager" Jul 29 15:55:02.503022 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.502944 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" containerName="manager" Jul 29 15:55:02.503022 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.503001 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7280f82-126b-4f8f-b2a1-934a40d46273" containerName="manager" Jul 29 15:55:02.503022 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.503011 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7feca122-2f3c-45df-9ac2-95414fcfe5ee" containerName="manager" Jul 29 15:55:02.504956 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.504939 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.509146 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.509117 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 15:55:02.509821 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.509803 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 15:55:02.509900 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.509851 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 15:55:02.509979 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.509957 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-52j2g\"" Jul 29 15:55:02.516201 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.516173 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpngp"] Jul 29 15:55:02.550233 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.550192 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bba9bd06-7fef-43d8-be21-e12b2d8d3610-config-file\") pod \"limitador-limitador-64c94fd64f-bpngp\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.550448 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.550308 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/bba9bd06-7fef-43d8-be21-e12b2d8d3610-kube-api-access-7wxg7\") pod \"limitador-limitador-64c94fd64f-bpngp\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.600761 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.600722 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpngp"] Jul 29 15:55:02.651017 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.650980 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/bba9bd06-7fef-43d8-be21-e12b2d8d3610-kube-api-access-7wxg7\") pod \"limitador-limitador-64c94fd64f-bpngp\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.651207 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.651044 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bba9bd06-7fef-43d8-be21-e12b2d8d3610-config-file\") pod \"limitador-limitador-64c94fd64f-bpngp\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.651701 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.651677 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bba9bd06-7fef-43d8-be21-e12b2d8d3610-config-file\") pod \"limitador-limitador-64c94fd64f-bpngp\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.659124 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.659101 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/bba9bd06-7fef-43d8-be21-e12b2d8d3610-kube-api-access-7wxg7\") pod \"limitador-limitador-64c94fd64f-bpngp\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.816638 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.816575 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:02.980596 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:02.980558 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpngp"] Jul 29 15:55:02.983259 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:55:02.983231 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba9bd06_7fef_43d8_be21_e12b2d8d3610.slice/crio-03bc7cc154dfd55163881d9267bb4003f88e5efc2973187b5e8124da37ab68f7 WatchSource:0}: Error finding container 03bc7cc154dfd55163881d9267bb4003f88e5efc2973187b5e8124da37ab68f7: Status 404 returned error can't find the container with id 03bc7cc154dfd55163881d9267bb4003f88e5efc2973187b5e8124da37ab68f7 Jul 29 15:55:03.346402 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:03.346364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" event={"ID":"bba9bd06-7fef-43d8-be21-e12b2d8d3610","Type":"ContainerStarted","Data":"03bc7cc154dfd55163881d9267bb4003f88e5efc2973187b5e8124da37ab68f7"} Jul 29 15:55:06.371322 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:06.371280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" event={"ID":"bba9bd06-7fef-43d8-be21-e12b2d8d3610","Type":"ContainerStarted","Data":"69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf"} Jul 29 15:55:06.371752 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:06.371446 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:06.390400 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:06.390341 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" podStartSLOduration=1.6005066000000001 podStartE2EDuration="4.390324476s" totalImagesPullingTime="2.789817876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:55:06.387528455 +0000 UTC m=+516.928429924" watchObservedRunningTime="2026-07-29 15:55:06.390324476 +0000 UTC m=+516.931225945" Jul 29 15:55:17.377007 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:17.376969 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:20.473871 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:20.473837 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpngp"] Jul 29 15:55:20.474275 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:20.474059 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" podUID="bba9bd06-7fef-43d8-be21-e12b2d8d3610" containerName="limitador" containerID="cri-o://69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf" gracePeriod=30 Jul 29 15:55:21.433618 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.433574 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:21.449500 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.449460 2555 generic.go:356] "Generic (PLEG): container finished" podID="bba9bd06-7fef-43d8-be21-e12b2d8d3610" containerID="69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf" exitCode=0 Jul 29 15:55:21.449689 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.449517 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" Jul 29 15:55:21.449689 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.449548 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" event={"ID":"bba9bd06-7fef-43d8-be21-e12b2d8d3610","Type":"ContainerDied","Data":"69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf"} Jul 29 15:55:21.449689 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.449647 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpngp" event={"ID":"bba9bd06-7fef-43d8-be21-e12b2d8d3610","Type":"ContainerDied","Data":"03bc7cc154dfd55163881d9267bb4003f88e5efc2973187b5e8124da37ab68f7"} Jul 29 15:55:21.449689 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.449670 2555 scope.go:117] "RemoveContainer" containerID="69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf" Jul 29 15:55:21.459976 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.459958 2555 scope.go:117] "RemoveContainer" containerID="69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf" Jul 29 15:55:21.460273 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:21.460252 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf\": container with ID starting with 69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf not found: ID does not exist" containerID="69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf" Jul 29 15:55:21.460337 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.460286 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf"} err="failed to get container status \"69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf\": rpc error: code = NotFound desc = could not find container \"69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf\": container with ID starting with 69dae28a795dc7c514cb28d7a55bfdd7bd8fbce7241e613518f24330e7021fbf not found: ID does not exist" Jul 29 15:55:21.521825 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.521785 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/bba9bd06-7fef-43d8-be21-e12b2d8d3610-kube-api-access-7wxg7\") pod \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " Jul 29 15:55:21.522260 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.521841 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bba9bd06-7fef-43d8-be21-e12b2d8d3610-config-file\") pod \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\" (UID: \"bba9bd06-7fef-43d8-be21-e12b2d8d3610\") " Jul 29 15:55:21.522260 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.522200 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba9bd06-7fef-43d8-be21-e12b2d8d3610-config-file" (OuterVolumeSpecName: "config-file") pod "bba9bd06-7fef-43d8-be21-e12b2d8d3610" (UID: "bba9bd06-7fef-43d8-be21-e12b2d8d3610"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:55:21.524005 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.523977 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba9bd06-7fef-43d8-be21-e12b2d8d3610-kube-api-access-7wxg7" (OuterVolumeSpecName: "kube-api-access-7wxg7") pod "bba9bd06-7fef-43d8-be21-e12b2d8d3610" (UID: "bba9bd06-7fef-43d8-be21-e12b2d8d3610"). InnerVolumeSpecName "kube-api-access-7wxg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:55:21.622710 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.622614 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/bba9bd06-7fef-43d8-be21-e12b2d8d3610-kube-api-access-7wxg7\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:55:21.622710 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.622647 2555 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bba9bd06-7fef-43d8-be21-e12b2d8d3610-config-file\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:55:21.774077 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.774043 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpngp"] Jul 29 15:55:21.779415 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.779388 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpngp"] Jul 29 15:55:21.924457 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:21.924382 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba9bd06-7fef-43d8-be21-e12b2d8d3610" path="/var/lib/kubelet/pods/bba9bd06-7fef-43d8-be21-e12b2d8d3610/volumes" Jul 29 15:55:24.326675 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.326638 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg"] Jul 29 15:55:24.327048 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.326952 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba9bd06-7fef-43d8-be21-e12b2d8d3610" containerName="limitador" Jul 29 15:55:24.327048 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.326963 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba9bd06-7fef-43d8-be21-e12b2d8d3610" containerName="limitador" Jul 29 15:55:24.327048 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.327032 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba9bd06-7fef-43d8-be21-e12b2d8d3610" containerName="limitador" Jul 29 15:55:24.328800 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.328785 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:24.332157 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:24.332126 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"postgres-69fb4594cf-5k6qg\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="008fc234-a584-490d-8faa-0abda8684e1d" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:24.332297 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:24.332161 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"postgres-creds\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" type="*v1.Secret" Jul 29 15:55:24.332297 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:24.332231 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 29 15:55:24.332714 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:24.332691 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 29 15:55:24.338331 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:24.338307 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-nnsqj\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nnsqj\"" type="*v1.Secret" Jul 29 15:55:24.349177 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.349150 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg"] Jul 29 15:55:24.448193 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.448158 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxw4\" (UniqueName: \"kubernetes.io/projected/008fc234-a584-490d-8faa-0abda8684e1d-kube-api-access-6jxw4\") pod \"postgres-69fb4594cf-5k6qg\" (UID: \"008fc234-a584-490d-8faa-0abda8684e1d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:24.448395 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.448297 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/008fc234-a584-490d-8faa-0abda8684e1d-data\") pod \"postgres-69fb4594cf-5k6qg\" (UID: \"008fc234-a584-490d-8faa-0abda8684e1d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:24.549076 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.549034 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/008fc234-a584-490d-8faa-0abda8684e1d-data\") pod \"postgres-69fb4594cf-5k6qg\" (UID: \"008fc234-a584-490d-8faa-0abda8684e1d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:24.549076 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.549080 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxw4\" (UniqueName: \"kubernetes.io/projected/008fc234-a584-490d-8faa-0abda8684e1d-kube-api-access-6jxw4\") pod \"postgres-69fb4594cf-5k6qg\" (UID: \"008fc234-a584-490d-8faa-0abda8684e1d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:24.549427 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:24.549405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/008fc234-a584-490d-8faa-0abda8684e1d-data\") pod \"postgres-69fb4594cf-5k6qg\" (UID: \"008fc234-a584-490d-8faa-0abda8684e1d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:25.467130 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:25.467101 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 15:55:25.698060 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:25.698021 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 15:55:25.718017 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:25.717943 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nnsqj\"" Jul 29 15:55:25.798712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:25.798676 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 15:55:25.801290 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:25.801255 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxw4\" (UniqueName: \"kubernetes.io/projected/008fc234-a584-490d-8faa-0abda8684e1d-kube-api-access-6jxw4\") pod \"postgres-69fb4594cf-5k6qg\" (UID: \"008fc234-a584-490d-8faa-0abda8684e1d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:25.839153 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:25.839117 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:25.972487 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:25.972462 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg"] Jul 29 15:55:25.975014 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:55:25.974983 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008fc234_a584_490d_8faa_0abda8684e1d.slice/crio-45c3e77b57fbc2cf5752adecc07b3f1d777606bf69e19aa90cd4de402bbfc6ca WatchSource:0}: Error finding container 45c3e77b57fbc2cf5752adecc07b3f1d777606bf69e19aa90cd4de402bbfc6ca: Status 404 returned error can't find the container with id 45c3e77b57fbc2cf5752adecc07b3f1d777606bf69e19aa90cd4de402bbfc6ca Jul 29 15:55:26.474491 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:26.474457 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" event={"ID":"008fc234-a584-490d-8faa-0abda8684e1d","Type":"ContainerStarted","Data":"45c3e77b57fbc2cf5752adecc07b3f1d777606bf69e19aa90cd4de402bbfc6ca"} Jul 29 15:55:30.764172 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:30.764149 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 15:55:31.504791 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:31.504758 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" event={"ID":"008fc234-a584-490d-8faa-0abda8684e1d","Type":"ContainerStarted","Data":"86de7e3f7e03fce12f87fbe464c911c5adef781a49096745dc3b80f7d26b2caa"} Jul 29 15:55:31.504977 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:31.504832 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:31.524000 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:31.523946 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" podStartSLOduration=2.7391139989999997 podStartE2EDuration="7.523927128s" totalImagesPullingTime="4.784813129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:55:31.52078028 +0000 UTC m=+542.061681749" watchObservedRunningTime="2026-07-29 15:55:31.523927128 +0000 UTC m=+542.064828599" Jul 29 15:55:37.541031 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:37.540997 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5k6qg" Jul 29 15:55:44.292400 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.292359 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-688b4886-hggdl"] Jul 29 15:55:44.313544 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.313509 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-688b4886-hggdl"] Jul 29 15:55:44.313750 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.313657 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:44.316576 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.316555 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 15:55:44.316813 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.316797 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 15:55:44.319961 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.319943 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vhf4m\"" Jul 29 15:55:44.320121 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.320085 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhgt\" (UniqueName: \"kubernetes.io/projected/fcd64548-51e5-420d-b492-0d67adbecaa4-kube-api-access-6nhgt\") pod \"maas-controller-688b4886-hggdl\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:44.320205 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.320186 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert\") pod \"maas-controller-688b4886-hggdl\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:44.421475 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.421434 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhgt\" (UniqueName: \"kubernetes.io/projected/fcd64548-51e5-420d-b492-0d67adbecaa4-kube-api-access-6nhgt\") pod \"maas-controller-688b4886-hggdl\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:44.421707 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.421487 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert\") pod \"maas-controller-688b4886-hggdl\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:44.421707 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:44.421628 2555 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 29 15:55:44.421707 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:44.421707 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert podName:fcd64548-51e5-420d-b492-0d67adbecaa4 nodeName:}" failed. No retries permitted until 2026-07-29 15:55:44.921691075 +0000 UTC m=+555.462592522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert") pod "maas-controller-688b4886-hggdl" (UID: "fcd64548-51e5-420d-b492-0d67adbecaa4") : secret "maas-controller-webhook-cert" not found Jul 29 15:55:44.433873 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.433845 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhgt\" (UniqueName: \"kubernetes.io/projected/fcd64548-51e5-420d-b492-0d67adbecaa4-kube-api-access-6nhgt\") pod \"maas-controller-688b4886-hggdl\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:44.925236 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.925197 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert\") pod \"maas-controller-688b4886-hggdl\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:44.927866 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:44.927839 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert\") pod \"maas-controller-688b4886-hggdl\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:45.224489 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:45.224378 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:45.359169 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:45.359142 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-688b4886-hggdl"] Jul 29 15:55:45.361308 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:55:45.361282 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd64548_51e5_420d_b492_0d67adbecaa4.slice/crio-3ed81db07781da351a69120e99ff646c05edb2f92727fe53b31a741e55ef3cf4 WatchSource:0}: Error finding container 3ed81db07781da351a69120e99ff646c05edb2f92727fe53b31a741e55ef3cf4: Status 404 returned error can't find the container with id 3ed81db07781da351a69120e99ff646c05edb2f92727fe53b31a741e55ef3cf4 Jul 29 15:55:45.574956 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:45.574916 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-688b4886-hggdl" event={"ID":"fcd64548-51e5-420d-b492-0d67adbecaa4","Type":"ContainerStarted","Data":"3ed81db07781da351a69120e99ff646c05edb2f92727fe53b31a741e55ef3cf4"} Jul 29 15:55:48.591872 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:48.591840 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-688b4886-hggdl" event={"ID":"fcd64548-51e5-420d-b492-0d67adbecaa4","Type":"ContainerStarted","Data":"df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59"} Jul 29 15:55:48.592268 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:48.591939 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:48.611753 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:48.611701 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-688b4886-hggdl" podStartSLOduration=2.330954577 podStartE2EDuration="4.611682628s" totalImagesPullingTime="2.280728051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:55:48.609703958 +0000 UTC m=+559.150605441" watchObservedRunningTime="2026-07-29 15:55:48.611682628 +0000 UTC m=+559.152584099" Jul 29 15:55:58.688623 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:58.688553 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-688b4886-hggdl"] Jul 29 15:55:58.689148 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:58.688826 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-688b4886-hggdl" podUID="fcd64548-51e5-420d-b492-0d67adbecaa4" containerName="manager" containerID="cri-o://df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59" gracePeriod=30 Jul 29 15:55:58.690780 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:58.690758 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:58.932315 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:58.932287 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:59.046638 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.046608 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert\") pod \"fcd64548-51e5-420d-b492-0d67adbecaa4\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " Jul 29 15:55:59.046807 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.046702 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nhgt\" (UniqueName: \"kubernetes.io/projected/fcd64548-51e5-420d-b492-0d67adbecaa4-kube-api-access-6nhgt\") pod \"fcd64548-51e5-420d-b492-0d67adbecaa4\" (UID: \"fcd64548-51e5-420d-b492-0d67adbecaa4\") " Jul 29 15:55:59.048799 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.048770 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fcd64548-51e5-420d-b492-0d67adbecaa4" (UID: "fcd64548-51e5-420d-b492-0d67adbecaa4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:55:59.048908 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.048810 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd64548-51e5-420d-b492-0d67adbecaa4-kube-api-access-6nhgt" (OuterVolumeSpecName: "kube-api-access-6nhgt") pod "fcd64548-51e5-420d-b492-0d67adbecaa4" (UID: "fcd64548-51e5-420d-b492-0d67adbecaa4"). InnerVolumeSpecName "kube-api-access-6nhgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:55:59.147304 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.147269 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcd64548-51e5-420d-b492-0d67adbecaa4-webhook-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:55:59.147304 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.147297 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhgt\" (UniqueName: \"kubernetes.io/projected/fcd64548-51e5-420d-b492-0d67adbecaa4-kube-api-access-6nhgt\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:55:59.648298 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.648269 2555 generic.go:356] "Generic (PLEG): container finished" podID="fcd64548-51e5-420d-b492-0d67adbecaa4" containerID="df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59" exitCode=0 Jul 29 15:55:59.648479 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.648315 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-688b4886-hggdl" Jul 29 15:55:59.648479 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.648355 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-688b4886-hggdl" event={"ID":"fcd64548-51e5-420d-b492-0d67adbecaa4","Type":"ContainerDied","Data":"df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59"} Jul 29 15:55:59.648479 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.648406 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-688b4886-hggdl" event={"ID":"fcd64548-51e5-420d-b492-0d67adbecaa4","Type":"ContainerDied","Data":"3ed81db07781da351a69120e99ff646c05edb2f92727fe53b31a741e55ef3cf4"} Jul 29 15:55:59.648479 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.648429 2555 scope.go:117] "RemoveContainer" containerID="df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59" Jul 29 15:55:59.658029 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.658015 2555 scope.go:117] "RemoveContainer" containerID="df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59" Jul 29 15:55:59.658272 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:55:59.658255 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59\": container with ID starting with df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59 not found: ID does not exist" containerID="df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59" Jul 29 15:55:59.658314 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.658281 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59"} err="failed to get container status \"df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59\": rpc error: code = NotFound desc = could not find container \"df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59\": container with ID starting with df5d724735d031a4ca5c3894925323a71206d13621e1232ddae03eecac64cb59 not found: ID does not exist" Jul 29 15:55:59.670999 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.670977 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-688b4886-hggdl"] Jul 29 15:55:59.674755 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.674735 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-688b4886-hggdl"] Jul 29 15:55:59.924877 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:55:59.924794 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcd64548-51e5-420d-b492-0d67adbecaa4" path="/var/lib/kubelet/pods/fcd64548-51e5-420d-b492-0d67adbecaa4/volumes" Jul 29 15:56:19.193089 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.193047 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79554c65f8-67w2q"] Jul 29 15:56:19.193729 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.193529 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd64548-51e5-420d-b492-0d67adbecaa4" containerName="manager" Jul 29 15:56:19.193729 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.193547 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd64548-51e5-420d-b492-0d67adbecaa4" containerName="manager" Jul 29 15:56:19.193729 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.193657 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd64548-51e5-420d-b492-0d67adbecaa4" containerName="manager" Jul 29 15:56:19.198159 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.198135 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.201114 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.201095 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 15:56:19.202134 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.202115 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqm6d\"" Jul 29 15:56:19.214495 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.214469 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79554c65f8-67w2q"] Jul 29 15:56:19.327421 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.327379 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-openshift-service-ca\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.327643 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.327430 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-maas-api-tls\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.327643 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.327538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnshw\" (UniqueName: \"kubernetes.io/projected/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-kube-api-access-hnshw\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.429001 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.428966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnshw\" (UniqueName: \"kubernetes.io/projected/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-kube-api-access-hnshw\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.429185 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.429025 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-openshift-service-ca\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.429185 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.429055 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-maas-api-tls\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.429766 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.429744 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-openshift-service-ca\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.431375 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.431355 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-maas-api-tls\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.437829 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.437807 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnshw\" (UniqueName: \"kubernetes.io/projected/efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7-kube-api-access-hnshw\") pod \"maas-api-79554c65f8-67w2q\" (UID: \"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7\") " pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.508377 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.508291 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:19.656443 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.656418 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79554c65f8-67w2q"] Jul 29 15:56:19.658535 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:56:19.658504 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa44fd7_5cf9_4b92_80cd_cc8ddecb0cf7.slice/crio-db7fe4b97daf06809ae029393bd2e36b727e2289b3a67d926a3e38bea858f187 WatchSource:0}: Error finding container db7fe4b97daf06809ae029393bd2e36b727e2289b3a67d926a3e38bea858f187: Status 404 returned error can't find the container with id db7fe4b97daf06809ae029393bd2e36b727e2289b3a67d926a3e38bea858f187 Jul 29 15:56:19.747652 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.747613 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" event={"ID":"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7","Type":"ContainerStarted","Data":"db7fe4b97daf06809ae029393bd2e36b727e2289b3a67d926a3e38bea858f187"} Jul 29 15:56:19.860485 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.860449 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl"] Jul 29 15:56:19.864885 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.864867 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:19.867935 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.867905 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 15:56:19.871137 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.871112 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hpvlw"] Jul 29 15:56:19.874469 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.874449 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:19.879012 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.878995 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tvc6j\"" Jul 29 15:56:19.885891 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.885867 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl"] Jul 29 15:56:19.914098 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.914064 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hpvlw"] Jul 29 15:56:19.933127 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.933084 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc2197-3336-4a4c-83d1-d7dd99c6d13f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-mvwtl\" (UID: \"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:19.933308 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:19.933159 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7qs\" (UniqueName: \"kubernetes.io/projected/1cbc2197-3336-4a4c-83d1-d7dd99c6d13f-kube-api-access-wz7qs\") pod \"payload-pre-processing-687dbb6cf6-mvwtl\" (UID: \"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:20.033639 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.033609 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlj9\" (UniqueName: \"kubernetes.io/projected/cc9cbd65-0255-4ed1-83ad-3d534ec9d42d-kube-api-access-jmlj9\") pod \"payload-processing-5cc6cd444b-hpvlw\" (UID: \"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:20.033840 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.033661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc2197-3336-4a4c-83d1-d7dd99c6d13f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-mvwtl\" (UID: \"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:20.033840 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.033680 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9cbd65-0255-4ed1-83ad-3d534ec9d42d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hpvlw\" (UID: \"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:20.033840 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.033731 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7qs\" (UniqueName: \"kubernetes.io/projected/1cbc2197-3336-4a4c-83d1-d7dd99c6d13f-kube-api-access-wz7qs\") pod \"payload-pre-processing-687dbb6cf6-mvwtl\" (UID: \"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:20.034455 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.034428 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc2197-3336-4a4c-83d1-d7dd99c6d13f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-mvwtl\" (UID: \"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:20.043272 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.043246 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7qs\" (UniqueName: \"kubernetes.io/projected/1cbc2197-3336-4a4c-83d1-d7dd99c6d13f-kube-api-access-wz7qs\") pod \"payload-pre-processing-687dbb6cf6-mvwtl\" (UID: \"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:20.134778 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.134685 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlj9\" (UniqueName: \"kubernetes.io/projected/cc9cbd65-0255-4ed1-83ad-3d534ec9d42d-kube-api-access-jmlj9\") pod \"payload-processing-5cc6cd444b-hpvlw\" (UID: \"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:20.134778 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.134736 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9cbd65-0255-4ed1-83ad-3d534ec9d42d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hpvlw\" (UID: \"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:20.135515 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.135491 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9cbd65-0255-4ed1-83ad-3d534ec9d42d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hpvlw\" (UID: \"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:20.143345 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.143324 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlj9\" (UniqueName: \"kubernetes.io/projected/cc9cbd65-0255-4ed1-83ad-3d534ec9d42d-kube-api-access-jmlj9\") pod \"payload-processing-5cc6cd444b-hpvlw\" (UID: \"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:20.176364 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.176327 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:20.183393 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.183368 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:20.526529 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.526496 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl"] Jul 29 15:56:20.529002 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:56:20.528761 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbc2197_3336_4a4c_83d1_d7dd99c6d13f.slice/crio-987cd0fd1a108c76da9f48f5898107778dfd2f5ba94dbd0c604cd6048b20ae31 WatchSource:0}: Error finding container 987cd0fd1a108c76da9f48f5898107778dfd2f5ba94dbd0c604cd6048b20ae31: Status 404 returned error can't find the container with id 987cd0fd1a108c76da9f48f5898107778dfd2f5ba94dbd0c604cd6048b20ae31 Jul 29 15:56:20.554609 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.554570 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hpvlw"] Jul 29 15:56:20.556865 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:56:20.556844 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9cbd65_0255_4ed1_83ad_3d534ec9d42d.slice/crio-593f03be1ac26ed0451059eaa15f77ec1aa70c5dfaf01ab965bad00d18dea14c WatchSource:0}: Error finding container 593f03be1ac26ed0451059eaa15f77ec1aa70c5dfaf01ab965bad00d18dea14c: Status 404 returned error can't find the container with id 593f03be1ac26ed0451059eaa15f77ec1aa70c5dfaf01ab965bad00d18dea14c Jul 29 15:56:20.755713 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.755678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" event={"ID":"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d","Type":"ContainerStarted","Data":"593f03be1ac26ed0451059eaa15f77ec1aa70c5dfaf01ab965bad00d18dea14c"} Jul 29 15:56:20.757070 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:20.757025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" event={"ID":"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f","Type":"ContainerStarted","Data":"987cd0fd1a108c76da9f48f5898107778dfd2f5ba94dbd0c604cd6048b20ae31"} Jul 29 15:56:21.767902 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:21.767864 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" event={"ID":"efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7","Type":"ContainerStarted","Data":"c08075af421238886802fde83f38d8fc196ade4530feb9f41222393d27053018"} Jul 29 15:56:21.768514 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:21.768183 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:21.789496 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:21.789430 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" podStartSLOduration=1.051465936 podStartE2EDuration="2.78940343s" totalImagesPullingTime="1.737937494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:56:21.785444901 +0000 UTC m=+592.326346370" watchObservedRunningTime="2026-07-29 15:56:21.78940343 +0000 UTC m=+592.330304879" Jul 29 15:56:22.774792 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:22.774753 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" event={"ID":"cc9cbd65-0255-4ed1-83ad-3d534ec9d42d","Type":"ContainerStarted","Data":"8d847aebc1da52b53496cd14cd3edf711ccb8483a4bf10e397eba2469fe1be2a"} Jul 29 15:56:22.775276 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:22.774981 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:22.776346 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:22.776309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" event={"ID":"1cbc2197-3336-4a4c-83d1-d7dd99c6d13f","Type":"ContainerStarted","Data":"e8700733d5b173af956fab57b6df8de924a76a5ff43d18738eca4fc3326a1bcd"} Jul 29 15:56:22.796548 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:22.796493 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" podStartSLOduration=1.751310287 podStartE2EDuration="3.796474356s" totalImagesPullingTime="2.045164069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:56:22.793886432 +0000 UTC m=+593.334787910" watchObservedRunningTime="2026-07-29 15:56:22.796474356 +0000 UTC m=+593.337375825" Jul 29 15:56:22.813444 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:22.813378 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" podStartSLOduration=1.751114139 podStartE2EDuration="3.813356829s" totalImagesPullingTime="2.06224269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:56:22.809743388 +0000 UTC m=+593.350644858" watchObservedRunningTime="2026-07-29 15:56:22.813356829 +0000 UTC m=+593.354258299" Jul 29 15:56:23.781020 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:23.780993 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:27.781807 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:27.781776 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79554c65f8-67w2q" Jul 29 15:56:29.868612 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:29.868566 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 15:56:29.870462 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:29.870436 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 15:56:33.783188 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:33.783148 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hpvlw" Jul 29 15:56:34.788717 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:34.788686 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-mvwtl" Jul 29 15:56:50.949405 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.949360 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s7287"] Jul 29 15:56:50.952855 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.952837 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:50.955701 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.955678 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 15:56:50.955855 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.955733 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 15:56:50.957190 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.957162 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j2td7\"" Jul 29 15:56:50.957313 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.957214 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 29 15:56:50.965284 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.965255 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s7287"] Jul 29 15:56:50.979818 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.979792 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28799df6-819e-480a-bccc-fdae883212a0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:50.980009 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.979842 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/28799df6-819e-480a-bccc-fdae883212a0-kube-api-access-2dmgb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:50.980009 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.979950 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:50.980009 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.979997 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:50.980179 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.980046 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:50.980179 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:50.980078 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081006 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.080964 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/28799df6-819e-480a-bccc-fdae883212a0-kube-api-access-2dmgb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081202 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081049 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081202 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081089 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081202 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081123 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081202 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081152 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081202 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081201 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28799df6-819e-480a-bccc-fdae883212a0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081546 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081520 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081651 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081524 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.081651 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.081566 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.083517 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.083486 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28799df6-819e-480a-bccc-fdae883212a0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.083702 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.083684 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28799df6-819e-480a-bccc-fdae883212a0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.089164 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.089144 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/28799df6-819e-480a-bccc-fdae883212a0-kube-api-access-2dmgb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s7287\" (UID: \"28799df6-819e-480a-bccc-fdae883212a0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.270133 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.270088 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:56:51.440139 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.440110 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s7287"] Jul 29 15:56:51.441479 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:56:51.441449 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28799df6_819e_480a_bccc_fdae883212a0.slice/crio-49f3aa718193c7b783831a110127c72964678d23325a5d93db63260979e86499 WatchSource:0}: Error finding container 49f3aa718193c7b783831a110127c72964678d23325a5d93db63260979e86499: Status 404 returned error can't find the container with id 49f3aa718193c7b783831a110127c72964678d23325a5d93db63260979e86499 Jul 29 15:56:51.442969 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.442948 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 15:56:51.926909 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:51.926865 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" event={"ID":"28799df6-819e-480a-bccc-fdae883212a0","Type":"ContainerStarted","Data":"49f3aa718193c7b783831a110127c72964678d23325a5d93db63260979e86499"} Jul 29 15:56:52.934036 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:52.933996 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" event={"ID":"28799df6-819e-480a-bccc-fdae883212a0","Type":"ContainerStarted","Data":"331b3de786a94fee24e16b61f01d71dd97b846a50514f0c43846cbf568df5c7c"} Jul 29 15:56:52.956255 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:56:52.956196 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" podStartSLOduration=1.965937255 podStartE2EDuration="2.956179928s" totalImagesPullingTime="990.242673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:56:52.95329919 +0000 UTC m=+623.494200660" watchObservedRunningTime="2026-07-29 15:56:52.956179928 +0000 UTC m=+623.497081396" Jul 29 15:57:01.271141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:01.271095 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:57:01.271646 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:01.271168 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:57:01.284141 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:01.284110 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:57:01.989564 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:01.989531 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s7287" Jul 29 15:57:03.348268 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.348229 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc"] Jul 29 15:57:03.351994 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.351974 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.354663 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.354640 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 29 15:57:03.364007 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.363978 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc"] Jul 29 15:57:03.389205 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.389172 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.389205 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.389208 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.389456 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.389225 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c83ed8a-e20c-4aac-a807-16724d4a9150-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.389456 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.389245 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsh2x\" (UniqueName: \"kubernetes.io/projected/0c83ed8a-e20c-4aac-a807-16724d4a9150-kube-api-access-qsh2x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.389456 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.389323 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.389456 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.389352 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.490399 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.490339 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.490399 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.490396 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.490640 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.490517 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c83ed8a-e20c-4aac-a807-16724d4a9150-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.490640 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.490563 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsh2x\" (UniqueName: \"kubernetes.io/projected/0c83ed8a-e20c-4aac-a807-16724d4a9150-kube-api-access-qsh2x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.490734 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.490640 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.490734 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.490664 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.490892 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.490864 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.491027 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.491008 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.491095 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.491013 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.492671 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.492649 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c83ed8a-e20c-4aac-a807-16724d4a9150-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.492857 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.492839 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c83ed8a-e20c-4aac-a807-16724d4a9150-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.498844 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.498818 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsh2x\" (UniqueName: \"kubernetes.io/projected/0c83ed8a-e20c-4aac-a807-16724d4a9150-kube-api-access-qsh2x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkpdc\" (UID: \"0c83ed8a-e20c-4aac-a807-16724d4a9150\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.663348 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.663239 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:03.803430 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.803399 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc"] Jul 29 15:57:03.805418 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:57:03.805380 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c83ed8a_e20c_4aac_a807_16724d4a9150.slice/crio-8e1f1132a90b4ebecc1d2849beb0fb7b7da8a6003c15c78ea1229d18b9e58b70 WatchSource:0}: Error finding container 8e1f1132a90b4ebecc1d2849beb0fb7b7da8a6003c15c78ea1229d18b9e58b70: Status 404 returned error can't find the container with id 8e1f1132a90b4ebecc1d2849beb0fb7b7da8a6003c15c78ea1229d18b9e58b70 Jul 29 15:57:03.990950 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:03.990856 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" event={"ID":"0c83ed8a-e20c-4aac-a807-16724d4a9150","Type":"ContainerStarted","Data":"8e1f1132a90b4ebecc1d2849beb0fb7b7da8a6003c15c78ea1229d18b9e58b70"} Jul 29 15:57:04.998944 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:04.998913 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" event={"ID":"0c83ed8a-e20c-4aac-a807-16724d4a9150","Type":"ContainerStarted","Data":"7052a80965bb50c379813c0ce3c9dc30260eb05c2e0455c21642c1119ce5eb84"} Jul 29 15:57:05.021456 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:05.021401 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" podStartSLOduration=1.775752614 podStartE2EDuration="2.021377862s" totalImagesPullingTime="245.625248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:57:05.018628943 +0000 UTC m=+635.559530411" watchObservedRunningTime="2026-07-29 15:57:05.021377862 +0000 UTC m=+635.562279330" Jul 29 15:57:13.663780 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:13.663746 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:13.664275 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:13.663818 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:13.676244 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:13.676219 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:57:14.057505 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:57:14.057472 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkpdc" Jul 29 15:58:44.648398 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.648305 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76bb745dd7-tm5qq"] Jul 29 15:58:44.652081 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.652054 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.674862 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.674826 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bb745dd7-tm5qq"] Jul 29 15:58:44.718239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.718197 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-serving-cert\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.718239 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.718239 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-oauth-serving-cert\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.718509 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.718338 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-config\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.718509 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.718377 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-service-ca\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.718509 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.718438 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-oauth-config\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.718509 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.718500 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-trusted-ca-bundle\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.718712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.718524 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6rz\" (UniqueName: \"kubernetes.io/projected/fc6ef919-cb3d-4918-9c30-a7c2319087ef-kube-api-access-gq6rz\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.819655 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.819575 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-config\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.819655 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.819655 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-service-ca\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.819875 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.819699 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-oauth-config\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.819875 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.819827 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-trusted-ca-bundle\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.819875 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.819862 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6rz\" (UniqueName: \"kubernetes.io/projected/fc6ef919-cb3d-4918-9c30-a7c2319087ef-kube-api-access-gq6rz\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.820009 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.819932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-serving-cert\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.820009 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.819975 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-oauth-serving-cert\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.820486 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.820459 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-service-ca\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.820658 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.820459 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-config\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.820740 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.820708 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-oauth-serving-cert\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.820804 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.820764 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6ef919-cb3d-4918-9c30-a7c2319087ef-trusted-ca-bundle\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.822268 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.822248 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-oauth-config\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.822370 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.822353 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6ef919-cb3d-4918-9c30-a7c2319087ef-console-serving-cert\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.832483 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.832453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6rz\" (UniqueName: \"kubernetes.io/projected/fc6ef919-cb3d-4918-9c30-a7c2319087ef-kube-api-access-gq6rz\") pod \"console-76bb745dd7-tm5qq\" (UID: \"fc6ef919-cb3d-4918-9c30-a7c2319087ef\") " pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:44.962004 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:44.961907 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:45.131734 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:45.131448 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bb745dd7-tm5qq"] Jul 29 15:58:45.134041 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:58:45.134009 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6ef919_cb3d_4918_9c30_a7c2319087ef.slice/crio-a2714de40169480d8433fd6d0a042a60e767a5ce5b43e04299d8b7e5561e446a WatchSource:0}: Error finding container a2714de40169480d8433fd6d0a042a60e767a5ce5b43e04299d8b7e5561e446a: Status 404 returned error can't find the container with id a2714de40169480d8433fd6d0a042a60e767a5ce5b43e04299d8b7e5561e446a Jul 29 15:58:45.512277 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:45.512237 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bb745dd7-tm5qq" event={"ID":"fc6ef919-cb3d-4918-9c30-a7c2319087ef","Type":"ContainerStarted","Data":"a77eae8159eccc177c104fa9a3f10974a82b207f49985dfbf6619e40f3ac805e"} Jul 29 15:58:45.512457 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:45.512288 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bb745dd7-tm5qq" event={"ID":"fc6ef919-cb3d-4918-9c30-a7c2319087ef","Type":"ContainerStarted","Data":"a2714de40169480d8433fd6d0a042a60e767a5ce5b43e04299d8b7e5561e446a"} Jul 29 15:58:45.545063 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:45.545010 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76bb745dd7-tm5qq" podStartSLOduration=1.544990482 podStartE2EDuration="1.544990482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:58:45.540389801 +0000 UTC m=+736.081291270" watchObservedRunningTime="2026-07-29 15:58:45.544990482 +0000 UTC m=+736.085891950" Jul 29 15:58:54.962262 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:54.962223 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:54.962841 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:54.962307 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:54.967295 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:54.967274 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:55.569089 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:55.569053 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76bb745dd7-tm5qq" Jul 29 15:58:55.616802 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:58:55.616764 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccdb6f58d-prvmk"] Jul 29 15:59:20.641531 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.641488 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ccdb6f58d-prvmk" podUID="17d66281-8aed-45bc-a9a2-d4010f5a6a9e" containerName="console" containerID="cri-o://3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8" gracePeriod=15 Jul 29 15:59:20.903998 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.903929 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:59:20.969537 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.969505 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-oauth-serving-cert\") pod \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " Jul 29 15:59:20.969712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.969558 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-config\") pod \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " Jul 29 15:59:20.969712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.969601 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-oauth-config\") pod \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " Jul 29 15:59:20.969712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.969620 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt269\" (UniqueName: \"kubernetes.io/projected/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-kube-api-access-qt269\") pod \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " Jul 29 15:59:20.969712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.969646 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-service-ca\") pod \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " Jul 29 15:59:20.969712 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.969677 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-trusted-ca-bundle\") pod \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " Jul 29 15:59:20.969993 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.969928 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-serving-cert\") pod \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\" (UID: \"17d66281-8aed-45bc-a9a2-d4010f5a6a9e\") " Jul 29 15:59:20.970085 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970059 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-config" (OuterVolumeSpecName: "console-config") pod "17d66281-8aed-45bc-a9a2-d4010f5a6a9e" (UID: "17d66281-8aed-45bc-a9a2-d4010f5a6a9e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:59:20.970175 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970149 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-service-ca" (OuterVolumeSpecName: "service-ca") pod "17d66281-8aed-45bc-a9a2-d4010f5a6a9e" (UID: "17d66281-8aed-45bc-a9a2-d4010f5a6a9e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:59:20.970230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970149 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17d66281-8aed-45bc-a9a2-d4010f5a6a9e" (UID: "17d66281-8aed-45bc-a9a2-d4010f5a6a9e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:59:20.970230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970162 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "17d66281-8aed-45bc-a9a2-d4010f5a6a9e" (UID: "17d66281-8aed-45bc-a9a2-d4010f5a6a9e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:59:20.970473 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970427 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:59:20.970473 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970452 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-trusted-ca-bundle\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:59:20.970473 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970467 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-oauth-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:59:20.970682 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.970480 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:59:20.971997 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.971966 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17d66281-8aed-45bc-a9a2-d4010f5a6a9e" (UID: "17d66281-8aed-45bc-a9a2-d4010f5a6a9e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:59:20.972111 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.972005 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-kube-api-access-qt269" (OuterVolumeSpecName: "kube-api-access-qt269") pod "17d66281-8aed-45bc-a9a2-d4010f5a6a9e" (UID: "17d66281-8aed-45bc-a9a2-d4010f5a6a9e"). InnerVolumeSpecName "kube-api-access-qt269". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:59:20.972111 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:20.972069 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17d66281-8aed-45bc-a9a2-d4010f5a6a9e" (UID: "17d66281-8aed-45bc-a9a2-d4010f5a6a9e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:59:21.071790 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.071751 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-serving-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:59:21.071790 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.071779 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-console-oauth-config\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:59:21.071790 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.071789 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt269\" (UniqueName: \"kubernetes.io/projected/17d66281-8aed-45bc-a9a2-d4010f5a6a9e-kube-api-access-qt269\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 15:59:21.698933 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.698892 2555 generic.go:356] "Generic (PLEG): container finished" podID="17d66281-8aed-45bc-a9a2-d4010f5a6a9e" containerID="3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8" exitCode=0 Jul 29 15:59:21.699418 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.698963 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccdb6f58d-prvmk" Jul 29 15:59:21.699418 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.698972 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb6f58d-prvmk" event={"ID":"17d66281-8aed-45bc-a9a2-d4010f5a6a9e","Type":"ContainerDied","Data":"3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8"} Jul 29 15:59:21.699418 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.699054 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccdb6f58d-prvmk" event={"ID":"17d66281-8aed-45bc-a9a2-d4010f5a6a9e","Type":"ContainerDied","Data":"c9c1daccacb0d13ea55b1becb25aba0722bdbc01ac4ec3ea392015f8f0b89ac7"} Jul 29 15:59:21.699418 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.699075 2555 scope.go:117] "RemoveContainer" containerID="3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8" Jul 29 15:59:21.708471 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.708452 2555 scope.go:117] "RemoveContainer" containerID="3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8" Jul 29 15:59:21.708803 ip-10-0-130-132 kubenswrapper[2555]: E0729 15:59:21.708783 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8\": container with ID starting with 3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8 not found: ID does not exist" containerID="3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8" Jul 29 15:59:21.708866 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.708812 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8"} err="failed to get container status \"3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8\": rpc error: code = NotFound desc = could not find container \"3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8\": container with ID starting with 3454b36e55dd81225a0edf6ef3bb6883b7e479e8a9b44a40484a685f06abf2f8 not found: ID does not exist" Jul 29 15:59:21.722619 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.722575 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccdb6f58d-prvmk"] Jul 29 15:59:21.726906 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.726878 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ccdb6f58d-prvmk"] Jul 29 15:59:21.813916 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.813871 2555 patch_prober.go:28] interesting pod/console-ccdb6f58d-prvmk container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.26:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 15:59:21.814081 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.813951 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-ccdb6f58d-prvmk" podUID="17d66281-8aed-45bc-a9a2-d4010f5a6a9e" containerName="console" probeResult="failure" output="Get \"https://10.132.0.26:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 29 15:59:21.924653 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:21.924621 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17d66281-8aed-45bc-a9a2-d4010f5a6a9e" path="/var/lib/kubelet/pods/17d66281-8aed-45bc-a9a2-d4010f5a6a9e/volumes" Jul 29 15:59:48.323645 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.323604 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7679986d7-9jwz7"] Jul 29 15:59:48.324196 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.324033 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17d66281-8aed-45bc-a9a2-d4010f5a6a9e" containerName="console" Jul 29 15:59:48.324196 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.324047 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17d66281-8aed-45bc-a9a2-d4010f5a6a9e" containerName="console" Jul 29 15:59:48.324196 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.324130 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="17d66281-8aed-45bc-a9a2-d4010f5a6a9e" containerName="console" Jul 29 15:59:48.328489 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.328472 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.331104 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.331082 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 15:59:48.332193 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.332177 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vhf4m\"" Jul 29 15:59:48.332268 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.332180 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 15:59:48.337230 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.337203 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7679986d7-9jwz7"] Jul 29 15:59:48.419929 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.419889 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c11fec76-a417-4ffa-99d6-7fcc9af430e9-webhook-cert\") pod \"maas-controller-7679986d7-9jwz7\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.420110 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.419965 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2hjc\" (UniqueName: \"kubernetes.io/projected/c11fec76-a417-4ffa-99d6-7fcc9af430e9-kube-api-access-q2hjc\") pod \"maas-controller-7679986d7-9jwz7\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.520761 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.520729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c11fec76-a417-4ffa-99d6-7fcc9af430e9-webhook-cert\") pod \"maas-controller-7679986d7-9jwz7\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.520939 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.520807 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2hjc\" (UniqueName: \"kubernetes.io/projected/c11fec76-a417-4ffa-99d6-7fcc9af430e9-kube-api-access-q2hjc\") pod \"maas-controller-7679986d7-9jwz7\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.523304 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.523271 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c11fec76-a417-4ffa-99d6-7fcc9af430e9-webhook-cert\") pod \"maas-controller-7679986d7-9jwz7\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.529514 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.529485 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2hjc\" (UniqueName: \"kubernetes.io/projected/c11fec76-a417-4ffa-99d6-7fcc9af430e9-kube-api-access-q2hjc\") pod \"maas-controller-7679986d7-9jwz7\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.640869 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.640776 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:48.782614 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.782573 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7679986d7-9jwz7"] Jul 29 15:59:48.784019 ip-10-0-130-132 kubenswrapper[2555]: W0729 15:59:48.783986 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11fec76_a417_4ffa_99d6_7fcc9af430e9.slice/crio-cac1b2b745f91b0d11db5e8a0bb82b1c947b84c87d0194281f4c5fe5e935b2d6 WatchSource:0}: Error finding container cac1b2b745f91b0d11db5e8a0bb82b1c947b84c87d0194281f4c5fe5e935b2d6: Status 404 returned error can't find the container with id cac1b2b745f91b0d11db5e8a0bb82b1c947b84c87d0194281f4c5fe5e935b2d6 Jul 29 15:59:48.844775 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:48.844737 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-9jwz7" event={"ID":"c11fec76-a417-4ffa-99d6-7fcc9af430e9","Type":"ContainerStarted","Data":"cac1b2b745f91b0d11db5e8a0bb82b1c947b84c87d0194281f4c5fe5e935b2d6"} Jul 29 15:59:49.853833 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:49.853791 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-9jwz7" event={"ID":"c11fec76-a417-4ffa-99d6-7fcc9af430e9","Type":"ContainerStarted","Data":"23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5"} Jul 29 15:59:49.854271 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:49.853865 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 15:59:49.873750 ip-10-0-130-132 kubenswrapper[2555]: I0729 15:59:49.873696 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7679986d7-9jwz7" podStartSLOduration=1.344546314 podStartE2EDuration="1.873674797s" totalImagesPullingTime="529.128483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:59:49.872078199 +0000 UTC m=+800.412979667" watchObservedRunningTime="2026-07-29 15:59:49.873674797 +0000 UTC m=+800.414576270" Jul 29 16:00:00.862170 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:00.862139 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 16:00:05.494145 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.494100 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7679986d7-9jwz7"] Jul 29 16:00:05.494614 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.494369 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7679986d7-9jwz7" podUID="c11fec76-a417-4ffa-99d6-7fcc9af430e9" containerName="manager" containerID="cri-o://23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5" gracePeriod=30 Jul 29 16:00:05.753281 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.753204 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 16:00:05.884912 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.884877 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c11fec76-a417-4ffa-99d6-7fcc9af430e9-webhook-cert\") pod \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " Jul 29 16:00:05.885102 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.884961 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2hjc\" (UniqueName: \"kubernetes.io/projected/c11fec76-a417-4ffa-99d6-7fcc9af430e9-kube-api-access-q2hjc\") pod \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\" (UID: \"c11fec76-a417-4ffa-99d6-7fcc9af430e9\") " Jul 29 16:00:05.886951 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.886918 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11fec76-a417-4ffa-99d6-7fcc9af430e9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c11fec76-a417-4ffa-99d6-7fcc9af430e9" (UID: "c11fec76-a417-4ffa-99d6-7fcc9af430e9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:00:05.887067 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.886981 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11fec76-a417-4ffa-99d6-7fcc9af430e9-kube-api-access-q2hjc" (OuterVolumeSpecName: "kube-api-access-q2hjc") pod "c11fec76-a417-4ffa-99d6-7fcc9af430e9" (UID: "c11fec76-a417-4ffa-99d6-7fcc9af430e9"). InnerVolumeSpecName "kube-api-access-q2hjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:00:05.940935 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.940900 2555 generic.go:356] "Generic (PLEG): container finished" podID="c11fec76-a417-4ffa-99d6-7fcc9af430e9" containerID="23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5" exitCode=0 Jul 29 16:00:05.940935 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.940938 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-9jwz7" Jul 29 16:00:05.941156 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.940969 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-9jwz7" event={"ID":"c11fec76-a417-4ffa-99d6-7fcc9af430e9","Type":"ContainerDied","Data":"23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5"} Jul 29 16:00:05.941156 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.941029 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-9jwz7" event={"ID":"c11fec76-a417-4ffa-99d6-7fcc9af430e9","Type":"ContainerDied","Data":"cac1b2b745f91b0d11db5e8a0bb82b1c947b84c87d0194281f4c5fe5e935b2d6"} Jul 29 16:00:05.941156 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.941053 2555 scope.go:117] "RemoveContainer" containerID="23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5" Jul 29 16:00:05.949990 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.949972 2555 scope.go:117] "RemoveContainer" containerID="23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5" Jul 29 16:00:05.950309 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:00:05.950287 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5\": container with ID starting with 23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5 not found: ID does not exist" containerID="23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5" Jul 29 16:00:05.950388 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.950318 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5"} err="failed to get container status \"23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5\": rpc error: code = NotFound desc = could not find container \"23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5\": container with ID starting with 23f132d7e204adad98c7cc7432e3693907a7a685a26fbcecf37b6b33518efeb5 not found: ID does not exist" Jul 29 16:00:05.959721 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.959690 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7679986d7-9jwz7"] Jul 29 16:00:05.961820 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.961799 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7679986d7-9jwz7"] Jul 29 16:00:05.986057 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.986022 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c11fec76-a417-4ffa-99d6-7fcc9af430e9-webhook-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:00:05.986057 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:05.986049 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2hjc\" (UniqueName: \"kubernetes.io/projected/c11fec76-a417-4ffa-99d6-7fcc9af430e9-kube-api-access-q2hjc\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:00:07.924397 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:00:07.924361 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c11fec76-a417-4ffa-99d6-7fcc9af430e9" path="/var/lib/kubelet/pods/c11fec76-a417-4ffa-99d6-7fcc9af430e9/volumes" Jul 29 16:01:29.900726 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:01:29.900648 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:01:29.904323 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:01:29.904300 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:06:29.933331 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:06:29.933303 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:06:29.937735 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:06:29.937709 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:11:29.965299 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:11:29.965266 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:11:29.971729 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:11:29.971704 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:13:48.473549 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.473456 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5"] Jul 29 16:13:48.478180 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.474976 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11fec76-a417-4ffa-99d6-7fcc9af430e9" containerName="manager" Jul 29 16:13:48.478180 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.475002 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11fec76-a417-4ffa-99d6-7fcc9af430e9" containerName="manager" Jul 29 16:13:48.478180 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.475163 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11fec76-a417-4ffa-99d6-7fcc9af430e9" containerName="manager" Jul 29 16:13:48.479020 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.478997 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.482021 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.481996 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2f353f9d-serving-cert\"" Jul 29 16:13:48.498284 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.498245 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5"] Jul 29 16:13:48.569547 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.569517 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.569739 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.569600 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e7f246-9839-41c1-8ea7-b7cce6402b55-openshift-service-ca\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.569739 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.569639 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xfvj\" (UniqueName: \"kubernetes.io/projected/30e7f246-9839-41c1-8ea7-b7cce6402b55-kube-api-access-7xfvj\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.670547 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.670501 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.670786 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.670613 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e7f246-9839-41c1-8ea7-b7cce6402b55-openshift-service-ca\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.670786 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.670661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xfvj\" (UniqueName: \"kubernetes.io/projected/30e7f246-9839-41c1-8ea7-b7cce6402b55-kube-api-access-7xfvj\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.670786 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:13:48.670681 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-serving-cert: secret "maas-api-e2e-ait-2f353f9d-serving-cert" not found Jul 29 16:13:48.670786 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:13:48.670757 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls podName:30e7f246-9839-41c1-8ea7-b7cce6402b55 nodeName:}" failed. No retries permitted until 2026-07-29 16:13:49.170739833 +0000 UTC m=+1639.711641280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls") pod "maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" (UID: "30e7f246-9839-41c1-8ea7-b7cce6402b55") : secret "maas-api-e2e-ait-2f353f9d-serving-cert" not found Jul 29 16:13:48.671219 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.671198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e7f246-9839-41c1-8ea7-b7cce6402b55-openshift-service-ca\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:48.684054 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:48.684023 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xfvj\" (UniqueName: \"kubernetes.io/projected/30e7f246-9839-41c1-8ea7-b7cce6402b55-kube-api-access-7xfvj\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:49.174229 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:49.174189 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:49.176835 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:49.176809 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls\") pod \"maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:49.389901 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:49.389858 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:49.539055 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:49.539025 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5"] Jul 29 16:13:49.541839 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:13:49.541791 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e7f246_9839_41c1_8ea7_b7cce6402b55.slice/crio-123a1c39c4e42dd8e2d69a14f8cae0b5a3a0978d1c299b0ee5b9b48bce6fde37 WatchSource:0}: Error finding container 123a1c39c4e42dd8e2d69a14f8cae0b5a3a0978d1c299b0ee5b9b48bce6fde37: Status 404 returned error can't find the container with id 123a1c39c4e42dd8e2d69a14f8cae0b5a3a0978d1c299b0ee5b9b48bce6fde37 Jul 29 16:13:49.543307 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:49.543286 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 16:13:50.153546 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:50.153514 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" event={"ID":"30e7f246-9839-41c1-8ea7-b7cce6402b55","Type":"ContainerStarted","Data":"2574f88c523c29d49d17af9252719d7c56d0db619ddae552fd9592eb28c52b01"} Jul 29 16:13:50.153757 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:50.153567 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" event={"ID":"30e7f246-9839-41c1-8ea7-b7cce6402b55","Type":"ContainerStarted","Data":"123a1c39c4e42dd8e2d69a14f8cae0b5a3a0978d1c299b0ee5b9b48bce6fde37"} Jul 29 16:13:50.153757 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:50.153605 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:13:50.178844 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:50.178785 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" podStartSLOduration=1.680757555 podStartE2EDuration="2.178764707s" totalImagesPullingTime="498.007152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:13:50.178350495 +0000 UTC m=+1640.719251976" watchObservedRunningTime="2026-07-29 16:13:50.178764707 +0000 UTC m=+1640.719666178" Jul 29 16:13:56.163626 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:13:56.163576 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:14:13.946210 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:13.946168 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64"] Jul 29 16:14:13.949663 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:13.949646 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" Jul 29 16:14:13.952448 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:13.952428 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:14:13.958856 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:13.958828 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64"] Jul 29 16:14:14.000093 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:14.000050 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a5b4af-5dd5-4ba1-9c0c-660a0115fac0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64\" (UID: \"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" Jul 29 16:14:14.101041 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:14.101000 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a5b4af-5dd5-4ba1-9c0c-660a0115fac0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64\" (UID: \"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" Jul 29 16:14:14.101776 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:14.101755 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a5b4af-5dd5-4ba1-9c0c-660a0115fac0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64\" (UID: \"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" Jul 29 16:14:14.261936 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:14.261905 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" Jul 29 16:14:14.402011 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:14.401976 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64"] Jul 29 16:14:14.403109 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:14:14.403080 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a5b4af_5dd5_4ba1_9c0c_660a0115fac0.slice/crio-2db396b44e03d61be978aeff9c420c9dffc2a70b36c98d1d232ae878586b2421 WatchSource:0}: Error finding container 2db396b44e03d61be978aeff9c420c9dffc2a70b36c98d1d232ae878586b2421: Status 404 returned error can't find the container with id 2db396b44e03d61be978aeff9c420c9dffc2a70b36c98d1d232ae878586b2421 Jul 29 16:14:15.297111 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:15.297078 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" event={"ID":"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0","Type":"ContainerStarted","Data":"2db396b44e03d61be978aeff9c420c9dffc2a70b36c98d1d232ae878586b2421"} Jul 29 16:14:17.310398 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:17.310364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" event={"ID":"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0","Type":"ContainerStarted","Data":"7292a7d0bdedf9e03771d0372afe5f61e82d7610b0e903b93263a85aa79c5e2d"} Jul 29 16:14:17.330278 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:17.330225 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" podStartSLOduration=2.105953347 podStartE2EDuration="4.330209025s" totalImagesPullingTime="2.224255678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:14:17.328252898 +0000 UTC m=+1667.869154358" watchObservedRunningTime="2026-07-29 16:14:17.330209025 +0000 UTC m=+1667.871110493" Jul 29 16:14:18.317666 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:18.317631 2555 generic.go:356] "Generic (PLEG): container finished" podID="27a5b4af-5dd5-4ba1-9c0c-660a0115fac0" containerID="7292a7d0bdedf9e03771d0372afe5f61e82d7610b0e903b93263a85aa79c5e2d" exitCode=0 Jul 29 16:14:18.318086 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:18.317726 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" event={"ID":"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0","Type":"ContainerDied","Data":"7292a7d0bdedf9e03771d0372afe5f61e82d7610b0e903b93263a85aa79c5e2d"} Jul 29 16:14:19.457395 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:19.457366 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" Jul 29 16:14:19.552002 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:19.551960 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a5b4af-5dd5-4ba1-9c0c-660a0115fac0-maas-api-service-ca\") pod \"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0\" (UID: \"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0\") " Jul 29 16:14:19.552326 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:19.552304 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a5b4af-5dd5-4ba1-9c0c-660a0115fac0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27a5b4af-5dd5-4ba1-9c0c-660a0115fac0" (UID: "27a5b4af-5dd5-4ba1-9c0c-660a0115fac0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:14:19.653373 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:19.653287 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a5b4af-5dd5-4ba1-9c0c-660a0115fac0-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:14:20.328826 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:20.328799 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" Jul 29 16:14:20.329006 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:20.328828 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64" event={"ID":"27a5b4af-5dd5-4ba1-9c0c-660a0115fac0","Type":"ContainerDied","Data":"2db396b44e03d61be978aeff9c420c9dffc2a70b36c98d1d232ae878586b2421"} Jul 29 16:14:20.329006 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:20.328879 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db396b44e03d61be978aeff9c420c9dffc2a70b36c98d1d232ae878586b2421" Jul 29 16:14:29.126462 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.126429 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5"] Jul 29 16:14:29.126936 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.126647 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" podUID="30e7f246-9839-41c1-8ea7-b7cce6402b55" containerName="maas-api" containerID="cri-o://2574f88c523c29d49d17af9252719d7c56d0db619ddae552fd9592eb28c52b01" gracePeriod=30 Jul 29 16:14:29.379306 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.379218 2555 generic.go:356] "Generic (PLEG): container finished" podID="30e7f246-9839-41c1-8ea7-b7cce6402b55" containerID="2574f88c523c29d49d17af9252719d7c56d0db619ddae552fd9592eb28c52b01" exitCode=0 Jul 29 16:14:29.379449 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.379296 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" event={"ID":"30e7f246-9839-41c1-8ea7-b7cce6402b55","Type":"ContainerDied","Data":"2574f88c523c29d49d17af9252719d7c56d0db619ddae552fd9592eb28c52b01"} Jul 29 16:14:29.379449 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.379357 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" event={"ID":"30e7f246-9839-41c1-8ea7-b7cce6402b55","Type":"ContainerDied","Data":"123a1c39c4e42dd8e2d69a14f8cae0b5a3a0978d1c299b0ee5b9b48bce6fde37"} Jul 29 16:14:29.379449 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.379379 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="123a1c39c4e42dd8e2d69a14f8cae0b5a3a0978d1c299b0ee5b9b48bce6fde37" Jul 29 16:14:29.381679 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.381658 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:14:29.387865 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.387838 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64"] Jul 29 16:14:29.389914 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.389894 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f353f9d-7xj64"] Jul 29 16:14:29.441785 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.441744 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls\") pod \"30e7f246-9839-41c1-8ea7-b7cce6402b55\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " Jul 29 16:14:29.441785 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.441794 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e7f246-9839-41c1-8ea7-b7cce6402b55-openshift-service-ca\") pod \"30e7f246-9839-41c1-8ea7-b7cce6402b55\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " Jul 29 16:14:29.442004 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.441813 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xfvj\" (UniqueName: \"kubernetes.io/projected/30e7f246-9839-41c1-8ea7-b7cce6402b55-kube-api-access-7xfvj\") pod \"30e7f246-9839-41c1-8ea7-b7cce6402b55\" (UID: \"30e7f246-9839-41c1-8ea7-b7cce6402b55\") " Jul 29 16:14:29.442156 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.442134 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e7f246-9839-41c1-8ea7-b7cce6402b55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30e7f246-9839-41c1-8ea7-b7cce6402b55" (UID: "30e7f246-9839-41c1-8ea7-b7cce6402b55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:14:29.444143 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.444120 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30e7f246-9839-41c1-8ea7-b7cce6402b55" (UID: "30e7f246-9839-41c1-8ea7-b7cce6402b55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:14:29.444218 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.444178 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e7f246-9839-41c1-8ea7-b7cce6402b55-kube-api-access-7xfvj" (OuterVolumeSpecName: "kube-api-access-7xfvj") pod "30e7f246-9839-41c1-8ea7-b7cce6402b55" (UID: "30e7f246-9839-41c1-8ea7-b7cce6402b55"). InnerVolumeSpecName "kube-api-access-7xfvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:14:29.543685 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.543655 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e7f246-9839-41c1-8ea7-b7cce6402b55-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:14:29.543685 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.543683 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e7f246-9839-41c1-8ea7-b7cce6402b55-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:14:29.543685 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.543693 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xfvj\" (UniqueName: \"kubernetes.io/projected/30e7f246-9839-41c1-8ea7-b7cce6402b55-kube-api-access-7xfvj\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:14:29.926687 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:29.926657 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a5b4af-5dd5-4ba1-9c0c-660a0115fac0" path="/var/lib/kubelet/pods/27a5b4af-5dd5-4ba1-9c0c-660a0115fac0/volumes" Jul 29 16:14:30.384789 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:30.384757 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5" Jul 29 16:14:30.412117 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:30.412079 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5"] Jul 29 16:14:30.419904 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:30.419872 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f353f9d-7f66b7dc49-54jf5"] Jul 29 16:14:31.929058 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:31.929024 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e7f246-9839-41c1-8ea7-b7cce6402b55" path="/var/lib/kubelet/pods/30e7f246-9839-41c1-8ea7-b7cce6402b55/volumes" Jul 29 16:14:33.795388 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.795351 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj"] Jul 29 16:14:33.795941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.795779 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e7f246-9839-41c1-8ea7-b7cce6402b55" containerName="maas-api" Jul 29 16:14:33.795941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.795791 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e7f246-9839-41c1-8ea7-b7cce6402b55" containerName="maas-api" Jul 29 16:14:33.795941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.795800 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a5b4af-5dd5-4ba1-9c0c-660a0115fac0" containerName="revoke-keys" Jul 29 16:14:33.795941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.795806 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a5b4af-5dd5-4ba1-9c0c-660a0115fac0" containerName="revoke-keys" Jul 29 16:14:33.795941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.795863 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e7f246-9839-41c1-8ea7-b7cce6402b55" containerName="maas-api" Jul 29 16:14:33.795941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.795870 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a5b4af-5dd5-4ba1-9c0c-660a0115fac0" containerName="revoke-keys" Jul 29 16:14:33.799149 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.799122 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.801803 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.801772 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-959c7923-serving-cert\"" Jul 29 16:14:33.812181 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.812148 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj"] Jul 29 16:14:33.881421 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.881373 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfp2\" (UniqueName: \"kubernetes.io/projected/b720c396-fa4a-49ca-9775-4fcc399067b7-kube-api-access-pcfp2\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.881421 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.881419 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b720c396-fa4a-49ca-9775-4fcc399067b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.881683 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.881542 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b720c396-fa4a-49ca-9775-4fcc399067b7-maas-api-tls\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.982187 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.982148 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfp2\" (UniqueName: \"kubernetes.io/projected/b720c396-fa4a-49ca-9775-4fcc399067b7-kube-api-access-pcfp2\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.982187 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.982185 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b720c396-fa4a-49ca-9775-4fcc399067b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.982405 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.982249 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b720c396-fa4a-49ca-9775-4fcc399067b7-maas-api-tls\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.983032 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.983006 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b720c396-fa4a-49ca-9775-4fcc399067b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.984711 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.984692 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b720c396-fa4a-49ca-9775-4fcc399067b7-maas-api-tls\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:33.992094 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:33.992067 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfp2\" (UniqueName: \"kubernetes.io/projected/b720c396-fa4a-49ca-9775-4fcc399067b7-kube-api-access-pcfp2\") pod \"maas-api-e2e-migrate-959c7923-5d666db969-4wgfj\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:34.110323 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:34.110234 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:34.256440 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:34.256414 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj"] Jul 29 16:14:34.258955 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:14:34.258923 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb720c396_fa4a_49ca_9775_4fcc399067b7.slice/crio-28e60963f2a902119cf83c8ec982ef91d6a71088666f85ef2e09bc379552a932 WatchSource:0}: Error finding container 28e60963f2a902119cf83c8ec982ef91d6a71088666f85ef2e09bc379552a932: Status 404 returned error can't find the container with id 28e60963f2a902119cf83c8ec982ef91d6a71088666f85ef2e09bc379552a932 Jul 29 16:14:34.406071 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:34.405984 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" event={"ID":"b720c396-fa4a-49ca-9775-4fcc399067b7","Type":"ContainerStarted","Data":"28e60963f2a902119cf83c8ec982ef91d6a71088666f85ef2e09bc379552a932"} Jul 29 16:14:35.412991 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:35.412947 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" event={"ID":"b720c396-fa4a-49ca-9775-4fcc399067b7","Type":"ContainerStarted","Data":"b3fed76c484a80732c9960861d147c3f237888315f2013b9f67cb7dc9311fce5"} Jul 29 16:14:35.413434 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:35.413013 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:14:35.432484 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:35.432416 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" podStartSLOduration=2.125452389 podStartE2EDuration="2.432391714s" totalImagesPullingTime="306.939325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:14:35.431445215 +0000 UTC m=+1685.972346684" watchObservedRunningTime="2026-07-29 16:14:35.432391714 +0000 UTC m=+1685.973293183" Jul 29 16:14:41.423825 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:14:41.423797 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:15:00.141616 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:00.141570 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf"] Jul 29 16:15:00.147640 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:00.147613 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf" Jul 29 16:15:00.151611 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:00.151566 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:15:00.157846 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:00.157814 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf"] Jul 29 16:15:00.161804 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:00.161773 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf" Jul 29 16:15:00.310098 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:00.310062 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf"] Jul 29 16:15:00.311256 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:15:00.311229 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca311f2_dded_4004_8f03_7f1e00a029a9.slice/crio-04c380c99ec499a8f93aadbb36962fd61ec83a1e473270e489245f9bdfcebb0a WatchSource:0}: Error finding container 04c380c99ec499a8f93aadbb36962fd61ec83a1e473270e489245f9bdfcebb0a: Status 404 returned error can't find the container with id 04c380c99ec499a8f93aadbb36962fd61ec83a1e473270e489245f9bdfcebb0a Jul 29 16:15:00.544049 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:00.544015 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf" event={"ID":"cca311f2-dded-4004-8f03-7f1e00a029a9","Type":"ContainerStarted","Data":"04c380c99ec499a8f93aadbb36962fd61ec83a1e473270e489245f9bdfcebb0a"} Jul 29 16:15:01.550378 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:01.550346 2555 generic.go:356] "Generic (PLEG): container finished" podID="cca311f2-dded-4004-8f03-7f1e00a029a9" containerID="8705657f820cdc0611a6944d651d49d34c4231991fa32434a4e66e6b8250f4db" exitCode=0 Jul 29 16:15:01.550784 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:01.550409 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf" event={"ID":"cca311f2-dded-4004-8f03-7f1e00a029a9","Type":"ContainerDied","Data":"8705657f820cdc0611a6944d651d49d34c4231991fa32434a4e66e6b8250f4db"} Jul 29 16:15:02.689996 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:02.689967 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf" Jul 29 16:15:03.571015 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:03.570975 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf" event={"ID":"cca311f2-dded-4004-8f03-7f1e00a029a9","Type":"ContainerDied","Data":"04c380c99ec499a8f93aadbb36962fd61ec83a1e473270e489245f9bdfcebb0a"} Jul 29 16:15:03.571191 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:03.571052 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c380c99ec499a8f93aadbb36962fd61ec83a1e473270e489245f9bdfcebb0a" Jul 29 16:15:03.571191 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:03.570987 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf" Jul 29 16:15:05.419346 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.419314 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj"] Jul 29 16:15:05.419901 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.419551 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" podUID="b720c396-fa4a-49ca-9775-4fcc399067b7" containerName="maas-api" containerID="cri-o://b3fed76c484a80732c9960861d147c3f237888315f2013b9f67cb7dc9311fce5" gracePeriod=30 Jul 29 16:15:05.450476 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.450445 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf"] Jul 29 16:15:05.454088 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.454056 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-959c7923-29755695-btvxf"] Jul 29 16:15:05.455681 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:15:05.455654 2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb720c396_fa4a_49ca_9775_4fcc399067b7.slice/crio-b3fed76c484a80732c9960861d147c3f237888315f2013b9f67cb7dc9311fce5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb720c396_fa4a_49ca_9775_4fcc399067b7.slice/crio-conmon-b3fed76c484a80732c9960861d147c3f237888315f2013b9f67cb7dc9311fce5.scope\": RecentStats: unable to find data in memory cache]" Jul 29 16:15:05.584496 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.584454 2555 generic.go:356] "Generic (PLEG): container finished" podID="b720c396-fa4a-49ca-9775-4fcc399067b7" containerID="b3fed76c484a80732c9960861d147c3f237888315f2013b9f67cb7dc9311fce5" exitCode=0 Jul 29 16:15:05.584677 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.584521 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" event={"ID":"b720c396-fa4a-49ca-9775-4fcc399067b7","Type":"ContainerDied","Data":"b3fed76c484a80732c9960861d147c3f237888315f2013b9f67cb7dc9311fce5"} Jul 29 16:15:05.682364 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.682284 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:15:05.768350 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.768298 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b720c396-fa4a-49ca-9775-4fcc399067b7-openshift-service-ca\") pod \"b720c396-fa4a-49ca-9775-4fcc399067b7\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " Jul 29 16:15:05.768350 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.768356 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b720c396-fa4a-49ca-9775-4fcc399067b7-maas-api-tls\") pod \"b720c396-fa4a-49ca-9775-4fcc399067b7\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " Jul 29 16:15:05.768574 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.768488 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcfp2\" (UniqueName: \"kubernetes.io/projected/b720c396-fa4a-49ca-9775-4fcc399067b7-kube-api-access-pcfp2\") pod \"b720c396-fa4a-49ca-9775-4fcc399067b7\" (UID: \"b720c396-fa4a-49ca-9775-4fcc399067b7\") " Jul 29 16:15:05.768774 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.768739 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b720c396-fa4a-49ca-9775-4fcc399067b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b720c396-fa4a-49ca-9775-4fcc399067b7" (UID: "b720c396-fa4a-49ca-9775-4fcc399067b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:15:05.770571 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.770538 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b720c396-fa4a-49ca-9775-4fcc399067b7-kube-api-access-pcfp2" (OuterVolumeSpecName: "kube-api-access-pcfp2") pod "b720c396-fa4a-49ca-9775-4fcc399067b7" (UID: "b720c396-fa4a-49ca-9775-4fcc399067b7"). InnerVolumeSpecName "kube-api-access-pcfp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:15:05.770571 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.770569 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b720c396-fa4a-49ca-9775-4fcc399067b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b720c396-fa4a-49ca-9775-4fcc399067b7" (UID: "b720c396-fa4a-49ca-9775-4fcc399067b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:15:05.869650 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.869606 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcfp2\" (UniqueName: \"kubernetes.io/projected/b720c396-fa4a-49ca-9775-4fcc399067b7-kube-api-access-pcfp2\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:15:05.869650 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.869643 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b720c396-fa4a-49ca-9775-4fcc399067b7-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:15:05.869650 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.869654 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b720c396-fa4a-49ca-9775-4fcc399067b7-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:15:05.924498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:05.924454 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca311f2-dded-4004-8f03-7f1e00a029a9" path="/var/lib/kubelet/pods/cca311f2-dded-4004-8f03-7f1e00a029a9/volumes" Jul 29 16:15:06.592508 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:06.592482 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" Jul 29 16:15:06.592947 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:06.592515 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj" event={"ID":"b720c396-fa4a-49ca-9775-4fcc399067b7","Type":"ContainerDied","Data":"28e60963f2a902119cf83c8ec982ef91d6a71088666f85ef2e09bc379552a932"} Jul 29 16:15:06.592947 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:06.592602 2555 scope.go:117] "RemoveContainer" containerID="b3fed76c484a80732c9960861d147c3f237888315f2013b9f67cb7dc9311fce5" Jul 29 16:15:06.666133 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:06.666095 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj"] Jul 29 16:15:06.683466 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:06.683428 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-959c7923-5d666db969-4wgfj"] Jul 29 16:15:07.924344 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:07.924309 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b720c396-fa4a-49ca-9775-4fcc399067b7" path="/var/lib/kubelet/pods/b720c396-fa4a-49ca-9775-4fcc399067b7/volumes" Jul 29 16:15:08.897666 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.897619 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq"] Jul 29 16:15:08.898203 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.898183 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b720c396-fa4a-49ca-9775-4fcc399067b7" containerName="maas-api" Jul 29 16:15:08.898288 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.898205 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b720c396-fa4a-49ca-9775-4fcc399067b7" containerName="maas-api" Jul 29 16:15:08.898288 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.898226 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca311f2-dded-4004-8f03-7f1e00a029a9" containerName="cleanup" Jul 29 16:15:08.898288 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.898235 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca311f2-dded-4004-8f03-7f1e00a029a9" containerName="cleanup" Jul 29 16:15:08.898381 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.898346 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b720c396-fa4a-49ca-9775-4fcc399067b7" containerName="maas-api" Jul 29 16:15:08.898381 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.898359 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca311f2-dded-4004-8f03-7f1e00a029a9" containerName="cleanup" Jul 29 16:15:08.902995 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.902971 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:08.905496 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.905476 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d49b3458-serving-cert\"" Jul 29 16:15:08.914912 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:08.914876 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq"] Jul 29 16:15:09.000911 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.000875 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f72b07d-5278-43d9-967d-3148baff6bcc-maas-api-tls\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.000911 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.000912 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f72b07d-5278-43d9-967d-3148baff6bcc-openshift-service-ca\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.001347 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.000933 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtb9j\" (UniqueName: \"kubernetes.io/projected/7f72b07d-5278-43d9-967d-3148baff6bcc-kube-api-access-jtb9j\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.102043 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.102007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f72b07d-5278-43d9-967d-3148baff6bcc-maas-api-tls\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.102043 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.102045 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f72b07d-5278-43d9-967d-3148baff6bcc-openshift-service-ca\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.102279 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.102072 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtb9j\" (UniqueName: \"kubernetes.io/projected/7f72b07d-5278-43d9-967d-3148baff6bcc-kube-api-access-jtb9j\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.102706 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.102687 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f72b07d-5278-43d9-967d-3148baff6bcc-openshift-service-ca\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.104554 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.104534 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f72b07d-5278-43d9-967d-3148baff6bcc-maas-api-tls\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.114824 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.114800 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtb9j\" (UniqueName: \"kubernetes.io/projected/7f72b07d-5278-43d9-967d-3148baff6bcc-kube-api-access-jtb9j\") pod \"maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.214524 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.214414 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:09.359976 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.359948 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq"] Jul 29 16:15:09.361606 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:15:09.361563 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f72b07d_5278_43d9_967d_3148baff6bcc.slice/crio-c16bcd61cdf692c16369a83cb8d90d552a39c516421e179a14404f72e9070cdd WatchSource:0}: Error finding container c16bcd61cdf692c16369a83cb8d90d552a39c516421e179a14404f72e9070cdd: Status 404 returned error can't find the container with id c16bcd61cdf692c16369a83cb8d90d552a39c516421e179a14404f72e9070cdd Jul 29 16:15:09.611308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:09.611261 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" event={"ID":"7f72b07d-5278-43d9-967d-3148baff6bcc","Type":"ContainerStarted","Data":"c16bcd61cdf692c16369a83cb8d90d552a39c516421e179a14404f72e9070cdd"} Jul 29 16:15:10.617483 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:10.617449 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" event={"ID":"7f72b07d-5278-43d9-967d-3148baff6bcc","Type":"ContainerStarted","Data":"8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f"} Jul 29 16:15:10.617917 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:10.617543 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:16.628736 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:16.628700 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:16.650453 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:16.650398 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" podStartSLOduration=8.196249107 podStartE2EDuration="8.650380646s" totalImagesPullingTime="454.131539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:15:10.63634634 +0000 UTC m=+1721.177247799" watchObservedRunningTime="2026-07-29 16:15:16.650380646 +0000 UTC m=+1727.191282115" Jul 29 16:15:40.439193 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.439156 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq"] Jul 29 16:15:40.439735 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.439376 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" podUID="7f72b07d-5278-43d9-967d-3148baff6bcc" containerName="maas-api" containerID="cri-o://8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f" gracePeriod=30 Jul 29 16:15:40.698525 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.698421 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:40.777145 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.777107 2555 generic.go:356] "Generic (PLEG): container finished" podID="7f72b07d-5278-43d9-967d-3148baff6bcc" containerID="8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f" exitCode=0 Jul 29 16:15:40.777356 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.777160 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" Jul 29 16:15:40.777356 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.777204 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" event={"ID":"7f72b07d-5278-43d9-967d-3148baff6bcc","Type":"ContainerDied","Data":"8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f"} Jul 29 16:15:40.777356 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.777289 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq" event={"ID":"7f72b07d-5278-43d9-967d-3148baff6bcc","Type":"ContainerDied","Data":"c16bcd61cdf692c16369a83cb8d90d552a39c516421e179a14404f72e9070cdd"} Jul 29 16:15:40.777356 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.777314 2555 scope.go:117] "RemoveContainer" containerID="8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f" Jul 29 16:15:40.786729 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.786708 2555 scope.go:117] "RemoveContainer" containerID="8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f" Jul 29 16:15:40.787032 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:15:40.787012 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f\": container with ID starting with 8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f not found: ID does not exist" containerID="8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f" Jul 29 16:15:40.787081 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.787049 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f"} err="failed to get container status \"8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f\": rpc error: code = NotFound desc = could not find container \"8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f\": container with ID starting with 8028afff01e87f5ae60dad627dd4d3e2ba9b59a328e8a6b3075a5777df59bc1f not found: ID does not exist" Jul 29 16:15:40.798638 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.798608 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f72b07d-5278-43d9-967d-3148baff6bcc-maas-api-tls\") pod \"7f72b07d-5278-43d9-967d-3148baff6bcc\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " Jul 29 16:15:40.798770 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.798670 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtb9j\" (UniqueName: \"kubernetes.io/projected/7f72b07d-5278-43d9-967d-3148baff6bcc-kube-api-access-jtb9j\") pod \"7f72b07d-5278-43d9-967d-3148baff6bcc\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " Jul 29 16:15:40.798847 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.798825 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f72b07d-5278-43d9-967d-3148baff6bcc-openshift-service-ca\") pod \"7f72b07d-5278-43d9-967d-3148baff6bcc\" (UID: \"7f72b07d-5278-43d9-967d-3148baff6bcc\") " Jul 29 16:15:40.799213 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.799181 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f72b07d-5278-43d9-967d-3148baff6bcc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f72b07d-5278-43d9-967d-3148baff6bcc" (UID: "7f72b07d-5278-43d9-967d-3148baff6bcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:15:40.800797 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.800775 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f72b07d-5278-43d9-967d-3148baff6bcc-kube-api-access-jtb9j" (OuterVolumeSpecName: "kube-api-access-jtb9j") pod "7f72b07d-5278-43d9-967d-3148baff6bcc" (UID: "7f72b07d-5278-43d9-967d-3148baff6bcc"). InnerVolumeSpecName "kube-api-access-jtb9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:15:40.800919 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.800901 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f72b07d-5278-43d9-967d-3148baff6bcc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f72b07d-5278-43d9-967d-3148baff6bcc" (UID: "7f72b07d-5278-43d9-967d-3148baff6bcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:15:40.899615 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.899546 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f72b07d-5278-43d9-967d-3148baff6bcc-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:15:40.899615 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.899576 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtb9j\" (UniqueName: \"kubernetes.io/projected/7f72b07d-5278-43d9-967d-3148baff6bcc-kube-api-access-jtb9j\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:15:40.899615 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:40.899619 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f72b07d-5278-43d9-967d-3148baff6bcc-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:15:41.103152 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:41.103110 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq"] Jul 29 16:15:41.110640 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:41.110611 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d49b3458-df88bd8cb-4hfcq"] Jul 29 16:15:41.925137 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:41.925104 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f72b07d-5278-43d9-967d-3148baff6bcc" path="/var/lib/kubelet/pods/7f72b07d-5278-43d9-967d-3148baff6bcc/volumes" Jul 29 16:15:45.084522 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.084482 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25"] Jul 29 16:15:45.084925 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.084901 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f72b07d-5278-43d9-967d-3148baff6bcc" containerName="maas-api" Jul 29 16:15:45.084925 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.084914 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f72b07d-5278-43d9-967d-3148baff6bcc" containerName="maas-api" Jul 29 16:15:45.084995 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.084969 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f72b07d-5278-43d9-967d-3148baff6bcc" containerName="maas-api" Jul 29 16:15:45.088482 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.088459 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.092004 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.091981 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-facc8562-serving-cert\"" Jul 29 16:15:45.102653 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.102619 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25"] Jul 29 16:15:45.238644 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.238605 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e673b2-6973-4950-9ec1-809f1615ed72-openshift-service-ca\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.238877 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.238655 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.238877 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.238716 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q7dw\" (UniqueName: \"kubernetes.io/projected/02e673b2-6973-4950-9ec1-809f1615ed72-kube-api-access-8q7dw\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.339678 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.339568 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e673b2-6973-4950-9ec1-809f1615ed72-openshift-service-ca\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.339678 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.339628 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.339678 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.339657 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q7dw\" (UniqueName: \"kubernetes.io/projected/02e673b2-6973-4950-9ec1-809f1615ed72-kube-api-access-8q7dw\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.339920 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:15:45.339802 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-serving-cert: secret "maas-api-e2e-derive-facc8562-serving-cert" not found Jul 29 16:15:45.339920 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:15:45.339895 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls podName:02e673b2-6973-4950-9ec1-809f1615ed72 nodeName:}" failed. No retries permitted until 2026-07-29 16:15:45.839870608 +0000 UTC m=+1756.380772081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls") pod "maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" (UID: "02e673b2-6973-4950-9ec1-809f1615ed72") : secret "maas-api-e2e-derive-facc8562-serving-cert" not found Jul 29 16:15:45.340369 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.340352 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e673b2-6973-4950-9ec1-809f1615ed72-openshift-service-ca\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.354090 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.354060 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q7dw\" (UniqueName: \"kubernetes.io/projected/02e673b2-6973-4950-9ec1-809f1615ed72-kube-api-access-8q7dw\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.845137 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.845096 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.847650 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.847627 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls\") pod \"maas-api-e2e-derive-facc8562-6c6f7494b-t9n25\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:45.998936 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:45.998897 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:46.146665 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:46.146640 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25"] Jul 29 16:15:46.152560 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:15:46.152208 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e673b2_6973_4950_9ec1_809f1615ed72.slice/crio-7cba2406d512cadb7e1439ebc3687c88f57e5ff2099e86bb9d98fa64585f8030 WatchSource:0}: Error finding container 7cba2406d512cadb7e1439ebc3687c88f57e5ff2099e86bb9d98fa64585f8030: Status 404 returned error can't find the container with id 7cba2406d512cadb7e1439ebc3687c88f57e5ff2099e86bb9d98fa64585f8030 Jul 29 16:15:46.810936 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:46.810901 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" event={"ID":"02e673b2-6973-4950-9ec1-809f1615ed72","Type":"ContainerStarted","Data":"727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e"} Jul 29 16:15:46.811119 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:46.810952 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" event={"ID":"02e673b2-6973-4950-9ec1-809f1615ed72","Type":"ContainerStarted","Data":"7cba2406d512cadb7e1439ebc3687c88f57e5ff2099e86bb9d98fa64585f8030"} Jul 29 16:15:46.811119 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:46.811011 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:15:46.830840 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:46.830769 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" podStartSLOduration=1.475777637 podStartE2EDuration="1.830746824s" totalImagesPullingTime="354.969187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:15:46.829794291 +0000 UTC m=+1757.370695772" watchObservedRunningTime="2026-07-29 16:15:46.830746824 +0000 UTC m=+1757.371648297" Jul 29 16:15:52.822833 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:15:52.822802 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:16:15.976100 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:15.976067 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25"] Jul 29 16:16:15.976659 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:15.976290 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" podUID="02e673b2-6973-4950-9ec1-809f1615ed72" containerName="maas-api" containerID="cri-o://727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e" gracePeriod=30 Jul 29 16:16:16.232625 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.232543 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:16:16.305645 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.305617 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q7dw\" (UniqueName: \"kubernetes.io/projected/02e673b2-6973-4950-9ec1-809f1615ed72-kube-api-access-8q7dw\") pod \"02e673b2-6973-4950-9ec1-809f1615ed72\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " Jul 29 16:16:16.305829 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.305662 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls\") pod \"02e673b2-6973-4950-9ec1-809f1615ed72\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " Jul 29 16:16:16.305829 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.305786 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e673b2-6973-4950-9ec1-809f1615ed72-openshift-service-ca\") pod \"02e673b2-6973-4950-9ec1-809f1615ed72\" (UID: \"02e673b2-6973-4950-9ec1-809f1615ed72\") " Jul 29 16:16:16.306295 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.306250 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e673b2-6973-4950-9ec1-809f1615ed72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02e673b2-6973-4950-9ec1-809f1615ed72" (UID: "02e673b2-6973-4950-9ec1-809f1615ed72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:16:16.308843 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.308819 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02e673b2-6973-4950-9ec1-809f1615ed72" (UID: "02e673b2-6973-4950-9ec1-809f1615ed72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:16:16.308973 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.308884 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e673b2-6973-4950-9ec1-809f1615ed72-kube-api-access-8q7dw" (OuterVolumeSpecName: "kube-api-access-8q7dw") pod "02e673b2-6973-4950-9ec1-809f1615ed72" (UID: "02e673b2-6973-4950-9ec1-809f1615ed72"). InnerVolumeSpecName "kube-api-access-8q7dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:16:16.407503 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.407467 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e673b2-6973-4950-9ec1-809f1615ed72-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:16:16.407503 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.407503 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e673b2-6973-4950-9ec1-809f1615ed72-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:16:16.407746 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.407518 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q7dw\" (UniqueName: \"kubernetes.io/projected/02e673b2-6973-4950-9ec1-809f1615ed72-kube-api-access-8q7dw\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:16:16.971889 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.971854 2555 generic.go:356] "Generic (PLEG): container finished" podID="02e673b2-6973-4950-9ec1-809f1615ed72" containerID="727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e" exitCode=0 Jul 29 16:16:16.972063 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.971897 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" Jul 29 16:16:16.972063 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.971929 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" event={"ID":"02e673b2-6973-4950-9ec1-809f1615ed72","Type":"ContainerDied","Data":"727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e"} Jul 29 16:16:16.972063 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.971973 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25" event={"ID":"02e673b2-6973-4950-9ec1-809f1615ed72","Type":"ContainerDied","Data":"7cba2406d512cadb7e1439ebc3687c88f57e5ff2099e86bb9d98fa64585f8030"} Jul 29 16:16:16.972063 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.971997 2555 scope.go:117] "RemoveContainer" containerID="727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e" Jul 29 16:16:16.984064 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.983759 2555 scope.go:117] "RemoveContainer" containerID="727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e" Jul 29 16:16:16.984511 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:16:16.984475 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e\": container with ID starting with 727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e not found: ID does not exist" containerID="727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e" Jul 29 16:16:16.984637 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:16.984514 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e"} err="failed to get container status \"727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e\": rpc error: code = NotFound desc = could not find container \"727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e\": container with ID starting with 727f741a1576872f0f31c5b59d7eee5d00149aca692ac1f9ad8004fc39d8d15e not found: ID does not exist" Jul 29 16:16:17.006235 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:17.006196 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25"] Jul 29 16:16:17.013659 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:17.013630 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-facc8562-6c6f7494b-t9n25"] Jul 29 16:16:17.924544 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:17.924512 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e673b2-6973-4950-9ec1-809f1615ed72" path="/var/lib/kubelet/pods/02e673b2-6973-4950-9ec1-809f1615ed72/volumes" Jul 29 16:16:29.997391 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:29.997365 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:16:30.005820 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:16:30.005795 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:18:10.613315 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.613280 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5"] Jul 29 16:18:10.613855 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.613687 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e673b2-6973-4950-9ec1-809f1615ed72" containerName="maas-api" Jul 29 16:18:10.613855 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.613699 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e673b2-6973-4950-9ec1-809f1615ed72" containerName="maas-api" Jul 29 16:18:10.613855 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.613754 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e673b2-6973-4950-9ec1-809f1615ed72" containerName="maas-api" Jul 29 16:18:10.616851 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.616831 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.620772 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.620749 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-82c422a4-serving-cert\"" Jul 29 16:18:10.659179 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.659146 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5"] Jul 29 16:18:10.727763 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.727726 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrswq\" (UniqueName: \"kubernetes.io/projected/734aa50e-5a92-40d4-8ae5-814bf9272225-kube-api-access-hrswq\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.727975 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.727776 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.727975 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.727844 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734aa50e-5a92-40d4-8ae5-814bf9272225-openshift-service-ca\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.829116 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.829057 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrswq\" (UniqueName: \"kubernetes.io/projected/734aa50e-5a92-40d4-8ae5-814bf9272225-kube-api-access-hrswq\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.829116 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.829123 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.829341 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.829144 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734aa50e-5a92-40d4-8ae5-814bf9272225-openshift-service-ca\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.829341 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:18:10.829288 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-serving-cert: secret "maas-api-e2e-mt-82c422a4-serving-cert" not found Jul 29 16:18:10.829429 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:18:10.829354 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls podName:734aa50e-5a92-40d4-8ae5-814bf9272225 nodeName:}" failed. No retries permitted until 2026-07-29 16:18:11.329337547 +0000 UTC m=+1901.870238994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls") pod "maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" (UID: "734aa50e-5a92-40d4-8ae5-814bf9272225") : secret "maas-api-e2e-mt-82c422a4-serving-cert" not found Jul 29 16:18:10.829781 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.829764 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734aa50e-5a92-40d4-8ae5-814bf9272225-openshift-service-ca\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:10.843824 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:10.843801 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrswq\" (UniqueName: \"kubernetes.io/projected/734aa50e-5a92-40d4-8ae5-814bf9272225-kube-api-access-hrswq\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:11.334430 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:11.334388 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:11.336983 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:11.336957 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls\") pod \"maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:11.528915 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:11.528873 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:11.699064 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:11.699038 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5"] Jul 29 16:18:11.701370 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:18:11.701342 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734aa50e_5a92_40d4_8ae5_814bf9272225.slice/crio-3b241dcabae8ff150ba759c68310bbefee9df9edfcc98d861d7629b707c0eb3c WatchSource:0}: Error finding container 3b241dcabae8ff150ba759c68310bbefee9df9edfcc98d861d7629b707c0eb3c: Status 404 returned error can't find the container with id 3b241dcabae8ff150ba759c68310bbefee9df9edfcc98d861d7629b707c0eb3c Jul 29 16:18:12.577731 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:12.577698 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" event={"ID":"734aa50e-5a92-40d4-8ae5-814bf9272225","Type":"ContainerStarted","Data":"e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe"} Jul 29 16:18:12.577937 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:12.577756 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" event={"ID":"734aa50e-5a92-40d4-8ae5-814bf9272225","Type":"ContainerStarted","Data":"3b241dcabae8ff150ba759c68310bbefee9df9edfcc98d861d7629b707c0eb3c"} Jul 29 16:18:12.577937 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:12.577811 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:12.601521 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:12.601462 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" podStartSLOduration=2.065146115 podStartE2EDuration="2.601441573s" totalImagesPullingTime="536.295458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:18:12.599541382 +0000 UTC m=+1903.140442852" watchObservedRunningTime="2026-07-29 16:18:12.601441573 +0000 UTC m=+1903.142343043" Jul 29 16:18:18.588944 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:18.588907 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:54.237938 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.237899 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5"] Jul 29 16:18:54.240344 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.238111 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" podUID="734aa50e-5a92-40d4-8ae5-814bf9272225" containerName="maas-api" containerID="cri-o://e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe" gracePeriod=30 Jul 29 16:18:54.500936 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.500869 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:54.626466 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.626429 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734aa50e-5a92-40d4-8ae5-814bf9272225-openshift-service-ca\") pod \"734aa50e-5a92-40d4-8ae5-814bf9272225\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " Jul 29 16:18:54.626689 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.626488 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrswq\" (UniqueName: \"kubernetes.io/projected/734aa50e-5a92-40d4-8ae5-814bf9272225-kube-api-access-hrswq\") pod \"734aa50e-5a92-40d4-8ae5-814bf9272225\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " Jul 29 16:18:54.626689 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.626652 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls\") pod \"734aa50e-5a92-40d4-8ae5-814bf9272225\" (UID: \"734aa50e-5a92-40d4-8ae5-814bf9272225\") " Jul 29 16:18:54.626930 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.626904 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734aa50e-5a92-40d4-8ae5-814bf9272225-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "734aa50e-5a92-40d4-8ae5-814bf9272225" (UID: "734aa50e-5a92-40d4-8ae5-814bf9272225"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:18:54.626991 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.626979 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734aa50e-5a92-40d4-8ae5-814bf9272225-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:18:54.628729 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.628702 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734aa50e-5a92-40d4-8ae5-814bf9272225-kube-api-access-hrswq" (OuterVolumeSpecName: "kube-api-access-hrswq") pod "734aa50e-5a92-40d4-8ae5-814bf9272225" (UID: "734aa50e-5a92-40d4-8ae5-814bf9272225"). InnerVolumeSpecName "kube-api-access-hrswq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:18:54.628729 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.628718 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "734aa50e-5a92-40d4-8ae5-814bf9272225" (UID: "734aa50e-5a92-40d4-8ae5-814bf9272225"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:18:54.728058 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.728020 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrswq\" (UniqueName: \"kubernetes.io/projected/734aa50e-5a92-40d4-8ae5-814bf9272225-kube-api-access-hrswq\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:18:54.728058 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.728056 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734aa50e-5a92-40d4-8ae5-814bf9272225-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:18:54.798811 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.798777 2555 generic.go:356] "Generic (PLEG): container finished" podID="734aa50e-5a92-40d4-8ae5-814bf9272225" containerID="e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe" exitCode=0 Jul 29 16:18:54.799020 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.798821 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" Jul 29 16:18:54.799020 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.798861 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" event={"ID":"734aa50e-5a92-40d4-8ae5-814bf9272225","Type":"ContainerDied","Data":"e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe"} Jul 29 16:18:54.799020 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.798925 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5" event={"ID":"734aa50e-5a92-40d4-8ae5-814bf9272225","Type":"ContainerDied","Data":"3b241dcabae8ff150ba759c68310bbefee9df9edfcc98d861d7629b707c0eb3c"} Jul 29 16:18:54.799020 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.798956 2555 scope.go:117] "RemoveContainer" containerID="e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe" Jul 29 16:18:54.808814 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.808792 2555 scope.go:117] "RemoveContainer" containerID="e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe" Jul 29 16:18:54.809117 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:18:54.809096 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe\": container with ID starting with e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe not found: ID does not exist" containerID="e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe" Jul 29 16:18:54.809191 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.809124 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe"} err="failed to get container status \"e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe\": rpc error: code = NotFound desc = could not find container \"e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe\": container with ID starting with e78eacfae4991008fd2720590e6d91582fdd23fe499fd44d74cf1a0d8389befe not found: ID does not exist" Jul 29 16:18:54.829179 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.829145 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5"] Jul 29 16:18:54.832091 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:54.832060 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82c422a4-749cd557d8-wqkq5"] Jul 29 16:18:55.924829 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:18:55.924796 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734aa50e-5a92-40d4-8ae5-814bf9272225" path="/var/lib/kubelet/pods/734aa50e-5a92-40d4-8ae5-814bf9272225/volumes" Jul 29 16:19:27.758303 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.758219 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n"] Jul 29 16:19:27.759037 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.758640 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734aa50e-5a92-40d4-8ae5-814bf9272225" containerName="maas-api" Jul 29 16:19:27.759037 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.758655 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="734aa50e-5a92-40d4-8ae5-814bf9272225" containerName="maas-api" Jul 29 16:19:27.759037 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.758727 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="734aa50e-5a92-40d4-8ae5-814bf9272225" containerName="maas-api" Jul 29 16:19:27.761840 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.761819 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.766025 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.766000 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c6a1f06a-serving-cert\"" Jul 29 16:19:27.775055 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.775030 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n"] Jul 29 16:19:27.822934 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.822895 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee57630-cd71-4a45-8cf8-39a30426a5a1-openshift-service-ca\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.822934 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.822942 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlnc9\" (UniqueName: \"kubernetes.io/projected/cee57630-cd71-4a45-8cf8-39a30426a5a1-kube-api-access-dlnc9\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.823205 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.823036 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.923498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.923467 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee57630-cd71-4a45-8cf8-39a30426a5a1-openshift-service-ca\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.923498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.923510 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlnc9\" (UniqueName: \"kubernetes.io/projected/cee57630-cd71-4a45-8cf8-39a30426a5a1-kube-api-access-dlnc9\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.923804 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.923599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.923804 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:19:27.923725 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-serving-cert: secret "maas-api-e2e-mt-c6a1f06a-serving-cert" not found Jul 29 16:19:27.923804 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:19:27.923798 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls podName:cee57630-cd71-4a45-8cf8-39a30426a5a1 nodeName:}" failed. No retries permitted until 2026-07-29 16:19:28.423777865 +0000 UTC m=+1978.964679328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls") pod "maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" (UID: "cee57630-cd71-4a45-8cf8-39a30426a5a1") : secret "maas-api-e2e-mt-c6a1f06a-serving-cert" not found Jul 29 16:19:27.924273 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.924250 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee57630-cd71-4a45-8cf8-39a30426a5a1-openshift-service-ca\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:27.939078 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:27.939044 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlnc9\" (UniqueName: \"kubernetes.io/projected/cee57630-cd71-4a45-8cf8-39a30426a5a1-kube-api-access-dlnc9\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:28.429163 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:28.429128 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:28.431668 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:28.431644 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls\") pod \"maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:28.671967 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:28.671926 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:28.822721 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:28.822687 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n"] Jul 29 16:19:28.824228 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:19:28.824186 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee57630_cd71_4a45_8cf8_39a30426a5a1.slice/crio-0d7676a9aba2696aac6440c31a9476818b23c6c975045111ac3bea5aa50d08fa WatchSource:0}: Error finding container 0d7676a9aba2696aac6440c31a9476818b23c6c975045111ac3bea5aa50d08fa: Status 404 returned error can't find the container with id 0d7676a9aba2696aac6440c31a9476818b23c6c975045111ac3bea5aa50d08fa Jul 29 16:19:28.825637 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:28.825616 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 16:19:28.989705 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:28.989614 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" event={"ID":"cee57630-cd71-4a45-8cf8-39a30426a5a1","Type":"ContainerStarted","Data":"0d7676a9aba2696aac6440c31a9476818b23c6c975045111ac3bea5aa50d08fa"} Jul 29 16:19:29.997654 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:29.997615 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" event={"ID":"cee57630-cd71-4a45-8cf8-39a30426a5a1","Type":"ContainerStarted","Data":"758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3"} Jul 29 16:19:29.998093 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:29.997724 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:19:30.017658 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:30.017598 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" podStartSLOduration=2.575601286 podStartE2EDuration="3.01756773s" totalImagesPullingTime="441.966444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:19:30.015999572 +0000 UTC m=+1980.556901042" watchObservedRunningTime="2026-07-29 16:19:30.01756773 +0000 UTC m=+1980.558469199" Jul 29 16:19:36.008170 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:19:36.008138 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:20:30.957426 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:30.957381 2555 scope.go:117] "RemoveContainer" containerID="2574f88c523c29d49d17af9252719d7c56d0db619ddae552fd9592eb28c52b01" Jul 29 16:20:30.966064 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:30.966040 2555 scope.go:117] "RemoveContainer" containerID="7292a7d0bdedf9e03771d0372afe5f61e82d7610b0e903b93263a85aa79c5e2d" Jul 29 16:20:47.780452 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:47.780417 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n"] Jul 29 16:20:47.781005 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:47.780649 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" podUID="cee57630-cd71-4a45-8cf8-39a30426a5a1" containerName="maas-api" containerID="cri-o://758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3" gracePeriod=30 Jul 29 16:20:48.038593 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.038499 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:20:48.173745 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.173705 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlnc9\" (UniqueName: \"kubernetes.io/projected/cee57630-cd71-4a45-8cf8-39a30426a5a1-kube-api-access-dlnc9\") pod \"cee57630-cd71-4a45-8cf8-39a30426a5a1\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " Jul 29 16:20:48.173917 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.173796 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee57630-cd71-4a45-8cf8-39a30426a5a1-openshift-service-ca\") pod \"cee57630-cd71-4a45-8cf8-39a30426a5a1\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " Jul 29 16:20:48.173917 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.173857 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls\") pod \"cee57630-cd71-4a45-8cf8-39a30426a5a1\" (UID: \"cee57630-cd71-4a45-8cf8-39a30426a5a1\") " Jul 29 16:20:48.174197 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.174163 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee57630-cd71-4a45-8cf8-39a30426a5a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cee57630-cd71-4a45-8cf8-39a30426a5a1" (UID: "cee57630-cd71-4a45-8cf8-39a30426a5a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:20:48.175997 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.175972 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee57630-cd71-4a45-8cf8-39a30426a5a1-kube-api-access-dlnc9" (OuterVolumeSpecName: "kube-api-access-dlnc9") pod "cee57630-cd71-4a45-8cf8-39a30426a5a1" (UID: "cee57630-cd71-4a45-8cf8-39a30426a5a1"). InnerVolumeSpecName "kube-api-access-dlnc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:20:48.176111 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.176073 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cee57630-cd71-4a45-8cf8-39a30426a5a1" (UID: "cee57630-cd71-4a45-8cf8-39a30426a5a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:20:48.274929 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.274891 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlnc9\" (UniqueName: \"kubernetes.io/projected/cee57630-cd71-4a45-8cf8-39a30426a5a1-kube-api-access-dlnc9\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:20:48.274929 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.274924 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee57630-cd71-4a45-8cf8-39a30426a5a1-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:20:48.274929 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.274934 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee57630-cd71-4a45-8cf8-39a30426a5a1-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:20:48.406712 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.406624 2555 generic.go:356] "Generic (PLEG): container finished" podID="cee57630-cd71-4a45-8cf8-39a30426a5a1" containerID="758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3" exitCode=0 Jul 29 16:20:48.406712 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.406674 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" Jul 29 16:20:48.406712 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.406701 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" event={"ID":"cee57630-cd71-4a45-8cf8-39a30426a5a1","Type":"ContainerDied","Data":"758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3"} Jul 29 16:20:48.406964 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.406758 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n" event={"ID":"cee57630-cd71-4a45-8cf8-39a30426a5a1","Type":"ContainerDied","Data":"0d7676a9aba2696aac6440c31a9476818b23c6c975045111ac3bea5aa50d08fa"} Jul 29 16:20:48.406964 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.406772 2555 scope.go:117] "RemoveContainer" containerID="758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3" Jul 29 16:20:48.419265 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.419243 2555 scope.go:117] "RemoveContainer" containerID="758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3" Jul 29 16:20:48.419614 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:20:48.419571 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3\": container with ID starting with 758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3 not found: ID does not exist" containerID="758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3" Jul 29 16:20:48.419727 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.419626 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3"} err="failed to get container status \"758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3\": rpc error: code = NotFound desc = could not find container \"758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3\": container with ID starting with 758374baa2a159bf4336bfc9119a9db4d166d686c4226a3eda2b6c80c235cef3 not found: ID does not exist" Jul 29 16:20:48.431487 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.431458 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n"] Jul 29 16:20:48.435429 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:48.435402 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6a1f06a-598c4bb7cb-sd78n"] Jul 29 16:20:49.924549 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:20:49.924517 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee57630-cd71-4a45-8cf8-39a30426a5a1" path="/var/lib/kubelet/pods/cee57630-cd71-4a45-8cf8-39a30426a5a1/volumes" Jul 29 16:21:30.029260 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:21:30.029227 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:21:30.038890 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:21:30.038863 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:21:30.989511 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:21:30.989470 2555 scope.go:117] "RemoveContainer" containerID="8705657f820cdc0611a6944d651d49d34c4231991fa32434a4e66e6b8250f4db" Jul 29 16:22:47.108013 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.107931 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq"] Jul 29 16:22:47.108433 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.108295 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee57630-cd71-4a45-8cf8-39a30426a5a1" containerName="maas-api" Jul 29 16:22:47.108433 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.108307 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee57630-cd71-4a45-8cf8-39a30426a5a1" containerName="maas-api" Jul 29 16:22:47.108433 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.108371 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee57630-cd71-4a45-8cf8-39a30426a5a1" containerName="maas-api" Jul 29 16:22:47.111349 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.111330 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.114771 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.114747 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b340d6-serving-cert\"" Jul 29 16:22:47.136717 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.136682 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq"] Jul 29 16:22:47.274172 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.274126 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.274380 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.274210 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.274380 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.274240 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npwn\" (UniqueName: \"kubernetes.io/projected/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-kube-api-access-4npwn\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.375462 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.375367 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.375462 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.375422 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npwn\" (UniqueName: \"kubernetes.io/projected/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-kube-api-access-4npwn\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.375689 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.375520 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.375689 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:22:47.375533 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-serving-cert: secret "maas-api-e2e-inf-a-b340d6-serving-cert" not found Jul 29 16:22:47.375689 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:22:47.375658 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls podName:c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b nodeName:}" failed. No retries permitted until 2026-07-29 16:22:47.875640127 +0000 UTC m=+2178.416541583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls") pod "maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" (UID: "c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b") : secret "maas-api-e2e-inf-a-b340d6-serving-cert" not found Jul 29 16:22:47.376136 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.376115 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.388489 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.388447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npwn\" (UniqueName: \"kubernetes.io/projected/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-kube-api-access-4npwn\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.881561 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.881522 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:47.883983 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:47.883950 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:48.022382 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:48.022345 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:48.161789 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:48.161765 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq"] Jul 29 16:22:48.163789 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:22:48.163754 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d8baa8_ac8f_4e05_aee5_680d4b1e5b1b.slice/crio-3bf87e23a3071d8f096c8b590ed83985e1f55f354737dd5a210bfc2cc58d731d WatchSource:0}: Error finding container 3bf87e23a3071d8f096c8b590ed83985e1f55f354737dd5a210bfc2cc58d731d: Status 404 returned error can't find the container with id 3bf87e23a3071d8f096c8b590ed83985e1f55f354737dd5a210bfc2cc58d731d Jul 29 16:22:49.023924 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:49.023815 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" event={"ID":"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b","Type":"ContainerStarted","Data":"a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3"} Jul 29 16:22:49.023924 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:49.023895 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:22:49.023924 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:49.023923 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" event={"ID":"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b","Type":"ContainerStarted","Data":"3bf87e23a3071d8f096c8b590ed83985e1f55f354737dd5a210bfc2cc58d731d"} Jul 29 16:22:49.041852 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:49.041801 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" podStartSLOduration=1.444179204 podStartE2EDuration="2.041783449s" totalImagesPullingTime="597.604245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:22:49.039749965 +0000 UTC m=+2179.580651435" watchObservedRunningTime="2026-07-29 16:22:49.041783449 +0000 UTC m=+2179.582684918" Jul 29 16:22:55.035489 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:22:55.035451 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:23:15.164520 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.164484 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm"] Jul 29 16:23:15.168181 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.168157 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.170838 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.170816 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8d3dc6-serving-cert\"" Jul 29 16:23:15.178379 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.178352 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm"] Jul 29 16:23:15.220684 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.220637 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca493e2-c379-438c-ae74-cd5af90460a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.220684 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.220687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk56p\" (UniqueName: \"kubernetes.io/projected/5ca493e2-c379-438c-ae74-cd5af90460a0-kube-api-access-vk56p\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.220907 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.220759 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.321697 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.321661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk56p\" (UniqueName: \"kubernetes.io/projected/5ca493e2-c379-438c-ae74-cd5af90460a0-kube-api-access-vk56p\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.321895 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.321729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.321895 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.321804 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca493e2-c379-438c-ae74-cd5af90460a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.321895 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:23:15.321845 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-serving-cert: secret "maas-api-e2e-inf-b-8d3dc6-serving-cert" not found Jul 29 16:23:15.322047 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:23:15.321914 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls podName:5ca493e2-c379-438c-ae74-cd5af90460a0 nodeName:}" failed. No retries permitted until 2026-07-29 16:23:15.821896403 +0000 UTC m=+2206.362797850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls") pod "maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" (UID: "5ca493e2-c379-438c-ae74-cd5af90460a0") : secret "maas-api-e2e-inf-b-8d3dc6-serving-cert" not found Jul 29 16:23:15.322411 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.322392 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca493e2-c379-438c-ae74-cd5af90460a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.332378 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.332346 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk56p\" (UniqueName: \"kubernetes.io/projected/5ca493e2-c379-438c-ae74-cd5af90460a0-kube-api-access-vk56p\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.826482 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.826437 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:15.829240 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:15.829162 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:16.080464 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:16.080368 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:16.221942 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:16.221917 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm"] Jul 29 16:23:16.223668 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:23:16.223635 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca493e2_c379_438c_ae74_cd5af90460a0.slice/crio-4c265e16166b86378f3803494bb0a0036502c310ef039b830324414094d1a902 WatchSource:0}: Error finding container 4c265e16166b86378f3803494bb0a0036502c310ef039b830324414094d1a902: Status 404 returned error can't find the container with id 4c265e16166b86378f3803494bb0a0036502c310ef039b830324414094d1a902 Jul 29 16:23:17.171271 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:17.171235 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" event={"ID":"5ca493e2-c379-438c-ae74-cd5af90460a0","Type":"ContainerStarted","Data":"fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11"} Jul 29 16:23:17.171497 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:17.171288 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" event={"ID":"5ca493e2-c379-438c-ae74-cd5af90460a0","Type":"ContainerStarted","Data":"4c265e16166b86378f3803494bb0a0036502c310ef039b830324414094d1a902"} Jul 29 16:23:17.171497 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:17.171346 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:23:17.211127 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:17.211068 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" podStartSLOduration=1.743473083 podStartE2EDuration="2.211050519s" totalImagesPullingTime="467.577436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:23:17.206227788 +0000 UTC m=+2207.747129257" watchObservedRunningTime="2026-07-29 16:23:17.211050519 +0000 UTC m=+2207.751951989" Jul 29 16:23:23.181765 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:23:23.181735 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:25:11.225282 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.225246 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq"] Jul 29 16:25:11.226408 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.226355 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" podUID="c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" containerName="maas-api" containerID="cri-o://a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3" gracePeriod=30 Jul 29 16:25:11.493882 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.493850 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:25:11.585311 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.585276 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-openshift-service-ca\") pod \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " Jul 29 16:25:11.585511 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.585349 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls\") pod \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " Jul 29 16:25:11.585511 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.585400 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4npwn\" (UniqueName: \"kubernetes.io/projected/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-kube-api-access-4npwn\") pod \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\" (UID: \"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b\") " Jul 29 16:25:11.585795 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.585762 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" (UID: "c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:25:11.587566 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.587539 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" (UID: "c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:25:11.587566 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.587551 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-kube-api-access-4npwn" (OuterVolumeSpecName: "kube-api-access-4npwn") pod "c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" (UID: "c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b"). InnerVolumeSpecName "kube-api-access-4npwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:25:11.686071 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.686023 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:25:11.686071 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.686062 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:25:11.686071 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.686075 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npwn\" (UniqueName: \"kubernetes.io/projected/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b-kube-api-access-4npwn\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:25:11.790451 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.790415 2555 generic.go:356] "Generic (PLEG): container finished" podID="c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" containerID="a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3" exitCode=0 Jul 29 16:25:11.790658 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.790464 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" Jul 29 16:25:11.790658 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.790503 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" event={"ID":"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b","Type":"ContainerDied","Data":"a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3"} Jul 29 16:25:11.790658 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.790579 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq" event={"ID":"c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b","Type":"ContainerDied","Data":"3bf87e23a3071d8f096c8b590ed83985e1f55f354737dd5a210bfc2cc58d731d"} Jul 29 16:25:11.790658 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.790626 2555 scope.go:117] "RemoveContainer" containerID="a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3" Jul 29 16:25:11.800762 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.800741 2555 scope.go:117] "RemoveContainer" containerID="a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3" Jul 29 16:25:11.801096 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:25:11.801073 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3\": container with ID starting with a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3 not found: ID does not exist" containerID="a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3" Jul 29 16:25:11.801162 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.801105 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3"} err="failed to get container status \"a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3\": rpc error: code = NotFound desc = could not find container \"a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3\": container with ID starting with a927cc25a0f401f7ecde340c88b1eb80e0f6517624dc1de652e714a778c304a3 not found: ID does not exist" Jul 29 16:25:11.813791 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.813752 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq"] Jul 29 16:25:11.817785 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.817757 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b340d6-66c4dfc46f-rccgq"] Jul 29 16:25:11.924373 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:11.924339 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" path="/var/lib/kubelet/pods/c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b/volumes" Jul 29 16:25:19.141076 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.141038 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92"] Jul 29 16:25:19.141486 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.141417 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" containerName="maas-api" Jul 29 16:25:19.141486 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.141428 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" containerName="maas-api" Jul 29 16:25:19.141563 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.141494 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d8baa8-ac8f-4e05-aee5-680d4b1e5b1b" containerName="maas-api" Jul 29 16:25:19.145743 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.145724 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" Jul 29 16:25:19.149545 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.149524 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:25:19.152235 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.152061 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe711bd6-9614-42a7-a3ad-9090c17b0ed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92\" (UID: \"fe711bd6-9614-42a7-a3ad-9090c17b0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" Jul 29 16:25:19.157760 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.157730 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92"] Jul 29 16:25:19.253041 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.253009 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe711bd6-9614-42a7-a3ad-9090c17b0ed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92\" (UID: \"fe711bd6-9614-42a7-a3ad-9090c17b0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" Jul 29 16:25:19.253663 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.253645 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe711bd6-9614-42a7-a3ad-9090c17b0ed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92\" (UID: \"fe711bd6-9614-42a7-a3ad-9090c17b0ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" Jul 29 16:25:19.455513 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.455415 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" Jul 29 16:25:19.609008 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.608942 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92"] Jul 29 16:25:19.610107 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:25:19.610084 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe711bd6_9614_42a7_a3ad_9090c17b0ed8.slice/crio-bf2d05c53dac1d22d59aaf3ed5bdc46c7903b4ef0fa752e03102a59aea2eddda WatchSource:0}: Error finding container bf2d05c53dac1d22d59aaf3ed5bdc46c7903b4ef0fa752e03102a59aea2eddda: Status 404 returned error can't find the container with id bf2d05c53dac1d22d59aaf3ed5bdc46c7903b4ef0fa752e03102a59aea2eddda Jul 29 16:25:19.840202 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:19.840168 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" event={"ID":"fe711bd6-9614-42a7-a3ad-9090c17b0ed8","Type":"ContainerStarted","Data":"bf2d05c53dac1d22d59aaf3ed5bdc46c7903b4ef0fa752e03102a59aea2eddda"} Jul 29 16:25:20.846303 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:20.846270 2555 generic.go:356] "Generic (PLEG): container finished" podID="fe711bd6-9614-42a7-a3ad-9090c17b0ed8" containerID="686119011e90afb17c0f3ee4d92b0b5b062dc54ddaf9002cde5b5fa9e7e0782b" exitCode=0 Jul 29 16:25:20.846754 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:20.846358 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" event={"ID":"fe711bd6-9614-42a7-a3ad-9090c17b0ed8","Type":"ContainerDied","Data":"686119011e90afb17c0f3ee4d92b0b5b062dc54ddaf9002cde5b5fa9e7e0782b"} Jul 29 16:25:21.987406 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:21.987379 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" Jul 29 16:25:22.072926 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:22.072888 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe711bd6-9614-42a7-a3ad-9090c17b0ed8-maas-api-service-ca\") pod \"fe711bd6-9614-42a7-a3ad-9090c17b0ed8\" (UID: \"fe711bd6-9614-42a7-a3ad-9090c17b0ed8\") " Jul 29 16:25:22.073298 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:22.073275 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe711bd6-9614-42a7-a3ad-9090c17b0ed8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe711bd6-9614-42a7-a3ad-9090c17b0ed8" (UID: "fe711bd6-9614-42a7-a3ad-9090c17b0ed8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:25:22.173989 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:22.173892 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe711bd6-9614-42a7-a3ad-9090c17b0ed8-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:25:22.857312 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:22.857284 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" Jul 29 16:25:22.857486 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:22.857309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92" event={"ID":"fe711bd6-9614-42a7-a3ad-9090c17b0ed8","Type":"ContainerDied","Data":"bf2d05c53dac1d22d59aaf3ed5bdc46c7903b4ef0fa752e03102a59aea2eddda"} Jul 29 16:25:22.857486 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:22.857352 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf2d05c53dac1d22d59aaf3ed5bdc46c7903b4ef0fa752e03102a59aea2eddda" Jul 29 16:25:23.942381 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:23.942349 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm"] Jul 29 16:25:23.942770 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:23.942557 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" podUID="5ca493e2-c379-438c-ae74-cd5af90460a0" containerName="maas-api" containerID="cri-o://fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11" gracePeriod=30 Jul 29 16:25:24.211599 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.211560 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:25:24.289403 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.289360 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca493e2-c379-438c-ae74-cd5af90460a0-openshift-service-ca\") pod \"5ca493e2-c379-438c-ae74-cd5af90460a0\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " Jul 29 16:25:24.289592 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.289412 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk56p\" (UniqueName: \"kubernetes.io/projected/5ca493e2-c379-438c-ae74-cd5af90460a0-kube-api-access-vk56p\") pod \"5ca493e2-c379-438c-ae74-cd5af90460a0\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " Jul 29 16:25:24.289592 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.289466 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls\") pod \"5ca493e2-c379-438c-ae74-cd5af90460a0\" (UID: \"5ca493e2-c379-438c-ae74-cd5af90460a0\") " Jul 29 16:25:24.289786 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.289757 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca493e2-c379-438c-ae74-cd5af90460a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ca493e2-c379-438c-ae74-cd5af90460a0" (UID: "5ca493e2-c379-438c-ae74-cd5af90460a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:25:24.291593 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.291553 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca493e2-c379-438c-ae74-cd5af90460a0-kube-api-access-vk56p" (OuterVolumeSpecName: "kube-api-access-vk56p") pod "5ca493e2-c379-438c-ae74-cd5af90460a0" (UID: "5ca493e2-c379-438c-ae74-cd5af90460a0"). InnerVolumeSpecName "kube-api-access-vk56p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:25:24.291729 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.291549 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ca493e2-c379-438c-ae74-cd5af90460a0" (UID: "5ca493e2-c379-438c-ae74-cd5af90460a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:25:24.370689 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.370649 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92"] Jul 29 16:25:24.377159 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.377130 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d3dc6-m7t92"] Jul 29 16:25:24.390420 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.390389 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca493e2-c379-438c-ae74-cd5af90460a0-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.390561 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.390424 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk56p\" (UniqueName: \"kubernetes.io/projected/5ca493e2-c379-438c-ae74-cd5af90460a0-kube-api-access-vk56p\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.390561 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.390442 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca493e2-c379-438c-ae74-cd5af90460a0-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.869105 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.869068 2555 generic.go:356] "Generic (PLEG): container finished" podID="5ca493e2-c379-438c-ae74-cd5af90460a0" containerID="fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11" exitCode=0 Jul 29 16:25:24.869300 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.869112 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" Jul 29 16:25:24.869300 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.869158 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" event={"ID":"5ca493e2-c379-438c-ae74-cd5af90460a0","Type":"ContainerDied","Data":"fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11"} Jul 29 16:25:24.869300 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.869214 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm" event={"ID":"5ca493e2-c379-438c-ae74-cd5af90460a0","Type":"ContainerDied","Data":"4c265e16166b86378f3803494bb0a0036502c310ef039b830324414094d1a902"} Jul 29 16:25:24.869300 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.869246 2555 scope.go:117] "RemoveContainer" containerID="fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11" Jul 29 16:25:24.879142 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.879123 2555 scope.go:117] "RemoveContainer" containerID="fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11" Jul 29 16:25:24.879490 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:25:24.879466 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11\": container with ID starting with fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11 not found: ID does not exist" containerID="fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11" Jul 29 16:25:24.879567 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.879504 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11"} err="failed to get container status \"fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11\": rpc error: code = NotFound desc = could not find container \"fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11\": container with ID starting with fc4233c9cd937bb20a0873e438b7cb4a72fba9a6672af5cc62bd2c8e6259fa11 not found: ID does not exist" Jul 29 16:25:24.903664 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.903631 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm"] Jul 29 16:25:24.908842 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:24.908812 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d3dc6-cbdd66cfc-trhsm"] Jul 29 16:25:25.924859 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:25.924826 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ca493e2-c379-438c-ae74-cd5af90460a0" path="/var/lib/kubelet/pods/5ca493e2-c379-438c-ae74-cd5af90460a0/volumes" Jul 29 16:25:25.925251 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:25.925177 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe711bd6-9614-42a7-a3ad-9090c17b0ed8" path="/var/lib/kubelet/pods/fe711bd6-9614-42a7-a3ad-9090c17b0ed8/volumes" Jul 29 16:25:35.898490 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.898454 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz"] Jul 29 16:25:35.898856 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.898822 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe711bd6-9614-42a7-a3ad-9090c17b0ed8" containerName="revoke-keys" Jul 29 16:25:35.898856 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.898834 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe711bd6-9614-42a7-a3ad-9090c17b0ed8" containerName="revoke-keys" Jul 29 16:25:35.898856 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.898853 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca493e2-c379-438c-ae74-cd5af90460a0" containerName="maas-api" Jul 29 16:25:35.898974 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.898859 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca493e2-c379-438c-ae74-cd5af90460a0" containerName="maas-api" Jul 29 16:25:35.898974 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.898925 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe711bd6-9614-42a7-a3ad-9090c17b0ed8" containerName="revoke-keys" Jul 29 16:25:35.898974 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.898936 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca493e2-c379-438c-ae74-cd5af90460a0" containerName="maas-api" Jul 29 16:25:35.901922 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.901907 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:35.904824 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.904799 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ef8cba-serving-cert\"" Jul 29 16:25:35.914700 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.914671 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz"] Jul 29 16:25:35.991955 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.991911 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7567276e-d7ee-4f68-aeea-8c61b909d393-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:35.992148 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.992074 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:35.992148 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:35.992112 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46htp\" (UniqueName: \"kubernetes.io/projected/7567276e-d7ee-4f68-aeea-8c61b909d393-kube-api-access-46htp\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.092963 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.092924 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.092963 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.092965 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46htp\" (UniqueName: \"kubernetes.io/projected/7567276e-d7ee-4f68-aeea-8c61b909d393-kube-api-access-46htp\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.093185 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.093009 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7567276e-d7ee-4f68-aeea-8c61b909d393-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.093185 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:25:36.093087 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-serving-cert: secret "maas-api-e2e-api-a-ef8cba-serving-cert" not found Jul 29 16:25:36.093257 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:25:36.093177 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls podName:7567276e-d7ee-4f68-aeea-8c61b909d393 nodeName:}" failed. No retries permitted until 2026-07-29 16:25:36.593155704 +0000 UTC m=+2347.134057165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls") pod "maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" (UID: "7567276e-d7ee-4f68-aeea-8c61b909d393") : secret "maas-api-e2e-api-a-ef8cba-serving-cert" not found Jul 29 16:25:36.093621 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.093607 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7567276e-d7ee-4f68-aeea-8c61b909d393-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.116834 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.116806 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46htp\" (UniqueName: \"kubernetes.io/projected/7567276e-d7ee-4f68-aeea-8c61b909d393-kube-api-access-46htp\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.598736 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.598689 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.601205 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.601182 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls\") pod \"maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.812375 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.812330 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:36.973031 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.972998 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz"] Jul 29 16:25:36.974326 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:25:36.974297 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7567276e_d7ee_4f68_aeea_8c61b909d393.slice/crio-67806a3994215dd066d0118ca81e7cb04c10b90de0fd0c62ba3b77c605dfb939 WatchSource:0}: Error finding container 67806a3994215dd066d0118ca81e7cb04c10b90de0fd0c62ba3b77c605dfb939: Status 404 returned error can't find the container with id 67806a3994215dd066d0118ca81e7cb04c10b90de0fd0c62ba3b77c605dfb939 Jul 29 16:25:36.975717 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:36.975696 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 16:25:37.942884 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:37.942840 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" event={"ID":"7567276e-d7ee-4f68-aeea-8c61b909d393","Type":"ContainerStarted","Data":"d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589"} Jul 29 16:25:37.943088 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:37.942913 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:25:37.943088 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:37.942938 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" event={"ID":"7567276e-d7ee-4f68-aeea-8c61b909d393","Type":"ContainerStarted","Data":"67806a3994215dd066d0118ca81e7cb04c10b90de0fd0c62ba3b77c605dfb939"} Jul 29 16:25:37.971310 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:37.971256 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" podStartSLOduration=2.5333143959999997 podStartE2EDuration="2.971239095s" totalImagesPullingTime="437.924699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:25:37.968898339 +0000 UTC m=+2348.509799855" watchObservedRunningTime="2026-07-29 16:25:37.971239095 +0000 UTC m=+2348.512140563" Jul 29 16:25:43.956013 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:25:43.955967 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:26:05.008846 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.008809 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf"] Jul 29 16:26:05.013049 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.013023 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.021373 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.021350 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6d4da6-serving-cert\"" Jul 29 16:26:05.027889 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.027860 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf"] Jul 29 16:26:05.168073 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.168038 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xszh4\" (UniqueName: \"kubernetes.io/projected/6b46a882-1866-4f61-9862-d67467dce134-kube-api-access-xszh4\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.168246 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.168090 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a882-1866-4f61-9862-d67467dce134-openshift-service-ca\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.168246 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.168203 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46a882-1866-4f61-9862-d67467dce134-maas-api-tls\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.269308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.269271 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xszh4\" (UniqueName: \"kubernetes.io/projected/6b46a882-1866-4f61-9862-d67467dce134-kube-api-access-xszh4\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.269488 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.269324 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a882-1866-4f61-9862-d67467dce134-openshift-service-ca\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.269488 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.269362 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46a882-1866-4f61-9862-d67467dce134-maas-api-tls\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.270046 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.270027 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a882-1866-4f61-9862-d67467dce134-openshift-service-ca\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.271774 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.271757 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46a882-1866-4f61-9862-d67467dce134-maas-api-tls\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.287662 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.287629 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xszh4\" (UniqueName: \"kubernetes.io/projected/6b46a882-1866-4f61-9862-d67467dce134-kube-api-access-xszh4\") pod \"maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.323315 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.323280 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:05.478939 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:05.478906 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf"] Jul 29 16:26:05.480496 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:26:05.480464 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b46a882_1866_4f61_9862_d67467dce134.slice/crio-fd0fb40271e2d9065a1202156d67ffd4e85dc28eb9c72607bb48049fb98bd91f WatchSource:0}: Error finding container fd0fb40271e2d9065a1202156d67ffd4e85dc28eb9c72607bb48049fb98bd91f: Status 404 returned error can't find the container with id fd0fb40271e2d9065a1202156d67ffd4e85dc28eb9c72607bb48049fb98bd91f Jul 29 16:26:06.091471 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:06.091432 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" event={"ID":"6b46a882-1866-4f61-9862-d67467dce134","Type":"ContainerStarted","Data":"11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631"} Jul 29 16:26:06.091920 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:06.091491 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" event={"ID":"6b46a882-1866-4f61-9862-d67467dce134","Type":"ContainerStarted","Data":"fd0fb40271e2d9065a1202156d67ffd4e85dc28eb9c72607bb48049fb98bd91f"} Jul 29 16:26:06.091920 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:06.091560 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:06.139560 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:06.139501 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" podStartSLOduration=1.6808532 podStartE2EDuration="2.139481962s" totalImagesPullingTime="458.628762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:26:06.137614677 +0000 UTC m=+2376.678516148" watchObservedRunningTime="2026-07-29 16:26:06.139481962 +0000 UTC m=+2376.680383442" Jul 29 16:26:12.102373 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:12.102344 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:26:30.061548 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:30.061511 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:26:30.074644 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:30.074619 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:26:48.641941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:48.641882 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz"] Jul 29 16:26:48.642564 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:48.642160 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" podUID="7567276e-d7ee-4f68-aeea-8c61b909d393" containerName="maas-api" containerID="cri-o://d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589" gracePeriod=30 Jul 29 16:26:48.893940 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:48.893870 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:26:49.083940 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.083906 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls\") pod \"7567276e-d7ee-4f68-aeea-8c61b909d393\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " Jul 29 16:26:49.084107 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.083952 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46htp\" (UniqueName: \"kubernetes.io/projected/7567276e-d7ee-4f68-aeea-8c61b909d393-kube-api-access-46htp\") pod \"7567276e-d7ee-4f68-aeea-8c61b909d393\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " Jul 29 16:26:49.084107 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.084032 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7567276e-d7ee-4f68-aeea-8c61b909d393-openshift-service-ca\") pod \"7567276e-d7ee-4f68-aeea-8c61b909d393\" (UID: \"7567276e-d7ee-4f68-aeea-8c61b909d393\") " Jul 29 16:26:49.084451 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.084423 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7567276e-d7ee-4f68-aeea-8c61b909d393-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7567276e-d7ee-4f68-aeea-8c61b909d393" (UID: "7567276e-d7ee-4f68-aeea-8c61b909d393"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:26:49.086017 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.085991 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7567276e-d7ee-4f68-aeea-8c61b909d393" (UID: "7567276e-d7ee-4f68-aeea-8c61b909d393"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:26:49.086123 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.086049 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7567276e-d7ee-4f68-aeea-8c61b909d393-kube-api-access-46htp" (OuterVolumeSpecName: "kube-api-access-46htp") pod "7567276e-d7ee-4f68-aeea-8c61b909d393" (UID: "7567276e-d7ee-4f68-aeea-8c61b909d393"). InnerVolumeSpecName "kube-api-access-46htp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:26:49.184844 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.184750 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7567276e-d7ee-4f68-aeea-8c61b909d393-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:26:49.184844 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.184783 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7567276e-d7ee-4f68-aeea-8c61b909d393-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:26:49.184844 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.184802 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46htp\" (UniqueName: \"kubernetes.io/projected/7567276e-d7ee-4f68-aeea-8c61b909d393-kube-api-access-46htp\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:26:49.317243 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.317211 2555 generic.go:356] "Generic (PLEG): container finished" podID="7567276e-d7ee-4f68-aeea-8c61b909d393" containerID="d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589" exitCode=0 Jul 29 16:26:49.317454 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.317256 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" Jul 29 16:26:49.317454 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.317292 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" event={"ID":"7567276e-d7ee-4f68-aeea-8c61b909d393","Type":"ContainerDied","Data":"d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589"} Jul 29 16:26:49.317454 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.317347 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz" event={"ID":"7567276e-d7ee-4f68-aeea-8c61b909d393","Type":"ContainerDied","Data":"67806a3994215dd066d0118ca81e7cb04c10b90de0fd0c62ba3b77c605dfb939"} Jul 29 16:26:49.317454 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.317377 2555 scope.go:117] "RemoveContainer" containerID="d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589" Jul 29 16:26:49.333853 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.333831 2555 scope.go:117] "RemoveContainer" containerID="d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589" Jul 29 16:26:49.334198 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:26:49.334177 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589\": container with ID starting with d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589 not found: ID does not exist" containerID="d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589" Jul 29 16:26:49.334247 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.334208 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589"} err="failed to get container status \"d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589\": rpc error: code = NotFound desc = could not find container \"d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589\": container with ID starting with d9a96120a79e68acd0fa994d64ec327fa1e47291fa012c5e0dbb2166a4b5c589 not found: ID does not exist" Jul 29 16:26:49.344880 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.344849 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz"] Jul 29 16:26:49.351041 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.351013 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8cba-696c6d6c9d-zrbkz"] Jul 29 16:26:49.924475 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:49.924442 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7567276e-d7ee-4f68-aeea-8c61b909d393" path="/var/lib/kubelet/pods/7567276e-d7ee-4f68-aeea-8c61b909d393/volumes" Jul 29 16:26:56.662960 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.662920 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f"] Jul 29 16:26:56.663336 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.663317 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7567276e-d7ee-4f68-aeea-8c61b909d393" containerName="maas-api" Jul 29 16:26:56.663336 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.663329 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7567276e-d7ee-4f68-aeea-8c61b909d393" containerName="maas-api" Jul 29 16:26:56.663426 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.663400 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7567276e-d7ee-4f68-aeea-8c61b909d393" containerName="maas-api" Jul 29 16:26:56.667861 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.667835 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" Jul 29 16:26:56.679719 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.679695 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:26:56.697730 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.697694 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f"] Jul 29 16:26:56.765940 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.765898 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86160b82-dd88-403f-b902-7b6bf98e374f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f\" (UID: \"86160b82-dd88-403f-b902-7b6bf98e374f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" Jul 29 16:26:56.866759 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.866721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86160b82-dd88-403f-b902-7b6bf98e374f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f\" (UID: \"86160b82-dd88-403f-b902-7b6bf98e374f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" Jul 29 16:26:56.867373 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.867350 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86160b82-dd88-403f-b902-7b6bf98e374f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f\" (UID: \"86160b82-dd88-403f-b902-7b6bf98e374f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" Jul 29 16:26:56.978150 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:56.978064 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" Jul 29 16:26:57.120188 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:57.120162 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f"] Jul 29 16:26:57.122188 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:26:57.122159 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86160b82_dd88_403f_b902_7b6bf98e374f.slice/crio-7465f84a3050ada3f68ee5258c85b7cd24f1800e15a82f64d80f824cfeb404e7 WatchSource:0}: Error finding container 7465f84a3050ada3f68ee5258c85b7cd24f1800e15a82f64d80f824cfeb404e7: Status 404 returned error can't find the container with id 7465f84a3050ada3f68ee5258c85b7cd24f1800e15a82f64d80f824cfeb404e7 Jul 29 16:26:57.368848 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:57.368814 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" event={"ID":"86160b82-dd88-403f-b902-7b6bf98e374f","Type":"ContainerStarted","Data":"7465f84a3050ada3f68ee5258c85b7cd24f1800e15a82f64d80f824cfeb404e7"} Jul 29 16:26:58.375156 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:58.375123 2555 generic.go:356] "Generic (PLEG): container finished" podID="86160b82-dd88-403f-b902-7b6bf98e374f" containerID="4a8847af58b688bd02ff1ff44ac9b48b1298f861e947dd1198b0ce588404e064" exitCode=0 Jul 29 16:26:58.375644 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:58.375212 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" event={"ID":"86160b82-dd88-403f-b902-7b6bf98e374f","Type":"ContainerDied","Data":"4a8847af58b688bd02ff1ff44ac9b48b1298f861e947dd1198b0ce588404e064"} Jul 29 16:26:59.521098 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:59.521071 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" Jul 29 16:26:59.591963 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:59.591926 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86160b82-dd88-403f-b902-7b6bf98e374f-maas-api-service-ca\") pod \"86160b82-dd88-403f-b902-7b6bf98e374f\" (UID: \"86160b82-dd88-403f-b902-7b6bf98e374f\") " Jul 29 16:26:59.592342 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:59.592314 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86160b82-dd88-403f-b902-7b6bf98e374f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86160b82-dd88-403f-b902-7b6bf98e374f" (UID: "86160b82-dd88-403f-b902-7b6bf98e374f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:26:59.693560 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:26:59.693461 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86160b82-dd88-403f-b902-7b6bf98e374f-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:27:00.387039 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:00.387002 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" Jul 29 16:27:00.387039 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:00.387031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f" event={"ID":"86160b82-dd88-403f-b902-7b6bf98e374f","Type":"ContainerDied","Data":"7465f84a3050ada3f68ee5258c85b7cd24f1800e15a82f64d80f824cfeb404e7"} Jul 29 16:27:00.387274 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:00.387089 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7465f84a3050ada3f68ee5258c85b7cd24f1800e15a82f64d80f824cfeb404e7" Jul 29 16:27:06.971220 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:06.971187 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf"] Jul 29 16:27:06.971797 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:06.971420 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" podUID="6b46a882-1866-4f61-9862-d67467dce134" containerName="maas-api" containerID="cri-o://11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631" gracePeriod=30 Jul 29 16:27:07.097573 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.097526 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" podUID="6b46a882-1866-4f61-9862-d67467dce134" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.61:8443/health\": dial tcp 10.132.0.61:8443: connect: connection refused" Jul 29 16:27:07.184545 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.184500 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f"] Jul 29 16:27:07.187345 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.187309 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6d4da6-45z6f"] Jul 29 16:27:07.234934 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.234860 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:27:07.363047 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.363005 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xszh4\" (UniqueName: \"kubernetes.io/projected/6b46a882-1866-4f61-9862-d67467dce134-kube-api-access-xszh4\") pod \"6b46a882-1866-4f61-9862-d67467dce134\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " Jul 29 16:27:07.363047 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.363047 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46a882-1866-4f61-9862-d67467dce134-maas-api-tls\") pod \"6b46a882-1866-4f61-9862-d67467dce134\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " Jul 29 16:27:07.363308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.363094 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a882-1866-4f61-9862-d67467dce134-openshift-service-ca\") pod \"6b46a882-1866-4f61-9862-d67467dce134\" (UID: \"6b46a882-1866-4f61-9862-d67467dce134\") " Jul 29 16:27:07.363490 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.363463 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b46a882-1866-4f61-9862-d67467dce134-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b46a882-1866-4f61-9862-d67467dce134" (UID: "6b46a882-1866-4f61-9862-d67467dce134"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:27:07.365284 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.365258 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b46a882-1866-4f61-9862-d67467dce134-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b46a882-1866-4f61-9862-d67467dce134" (UID: "6b46a882-1866-4f61-9862-d67467dce134"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:27:07.365364 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.365306 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b46a882-1866-4f61-9862-d67467dce134-kube-api-access-xszh4" (OuterVolumeSpecName: "kube-api-access-xszh4") pod "6b46a882-1866-4f61-9862-d67467dce134" (UID: "6b46a882-1866-4f61-9862-d67467dce134"). InnerVolumeSpecName "kube-api-access-xszh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:27:07.424031 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.423997 2555 generic.go:356] "Generic (PLEG): container finished" podID="6b46a882-1866-4f61-9862-d67467dce134" containerID="11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631" exitCode=0 Jul 29 16:27:07.424201 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.424046 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" Jul 29 16:27:07.424201 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.424074 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" event={"ID":"6b46a882-1866-4f61-9862-d67467dce134","Type":"ContainerDied","Data":"11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631"} Jul 29 16:27:07.424201 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.424117 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf" event={"ID":"6b46a882-1866-4f61-9862-d67467dce134","Type":"ContainerDied","Data":"fd0fb40271e2d9065a1202156d67ffd4e85dc28eb9c72607bb48049fb98bd91f"} Jul 29 16:27:07.424201 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.424141 2555 scope.go:117] "RemoveContainer" containerID="11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631" Jul 29 16:27:07.433287 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.433268 2555 scope.go:117] "RemoveContainer" containerID="11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631" Jul 29 16:27:07.433570 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:27:07.433550 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631\": container with ID starting with 11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631 not found: ID does not exist" containerID="11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631" Jul 29 16:27:07.433656 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.433599 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631"} err="failed to get container status \"11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631\": rpc error: code = NotFound desc = could not find container \"11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631\": container with ID starting with 11a65c893472650783c68d1e8d8e218bc8f3c818f6300e69fb369f746bde0631 not found: ID does not exist" Jul 29 16:27:07.452688 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.452600 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf"] Jul 29 16:27:07.454514 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.454479 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6d4da6-858b97fc59-4fxzf"] Jul 29 16:27:07.463801 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.463769 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xszh4\" (UniqueName: \"kubernetes.io/projected/6b46a882-1866-4f61-9862-d67467dce134-kube-api-access-xszh4\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:27:07.463801 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.463799 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46a882-1866-4f61-9862-d67467dce134-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:27:07.463962 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.463813 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46a882-1866-4f61-9862-d67467dce134-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:27:07.924121 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.924089 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b46a882-1866-4f61-9862-d67467dce134" path="/var/lib/kubelet/pods/6b46a882-1866-4f61-9862-d67467dce134/volumes" Jul 29 16:27:07.924432 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:07.924420 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86160b82-dd88-403f-b902-7b6bf98e374f" path="/var/lib/kubelet/pods/86160b82-dd88-403f-b902-7b6bf98e374f/volumes" Jul 29 16:27:18.374986 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.374946 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f"] Jul 29 16:27:18.375860 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.375831 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b46a882-1866-4f61-9862-d67467dce134" containerName="maas-api" Jul 29 16:27:18.375860 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.375858 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b46a882-1866-4f61-9862-d67467dce134" containerName="maas-api" Jul 29 16:27:18.376048 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.375872 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86160b82-dd88-403f-b902-7b6bf98e374f" containerName="revoke-keys" Jul 29 16:27:18.376048 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.375881 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86160b82-dd88-403f-b902-7b6bf98e374f" containerName="revoke-keys" Jul 29 16:27:18.376048 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.375984 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b46a882-1866-4f61-9862-d67467dce134" containerName="maas-api" Jul 29 16:27:18.376048 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.375999 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="86160b82-dd88-403f-b902-7b6bf98e374f" containerName="revoke-keys" Jul 29 16:27:18.380548 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.380523 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.386027 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.385997 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-81ab8b-serving-cert\"" Jul 29 16:27:18.392636 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.392573 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f"] Jul 29 16:27:18.464070 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.464030 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe54bf4-8094-4c13-8af6-24348f1e568b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.464070 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.464076 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc99g\" (UniqueName: \"kubernetes.io/projected/bbe54bf4-8094-4c13-8af6-24348f1e568b-kube-api-access-bc99g\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.464291 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.464102 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.565769 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.565729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe54bf4-8094-4c13-8af6-24348f1e568b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.565985 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.565779 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc99g\" (UniqueName: \"kubernetes.io/projected/bbe54bf4-8094-4c13-8af6-24348f1e568b-kube-api-access-bc99g\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.565985 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.565818 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.565985 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:27:18.565979 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-serving-cert: secret "maas-api-e2e-shared-a-81ab8b-serving-cert" not found Jul 29 16:27:18.566153 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:27:18.566073 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls podName:bbe54bf4-8094-4c13-8af6-24348f1e568b nodeName:}" failed. No retries permitted until 2026-07-29 16:27:19.066051538 +0000 UTC m=+2449.606952999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls") pod "maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" (UID: "bbe54bf4-8094-4c13-8af6-24348f1e568b") : secret "maas-api-e2e-shared-a-81ab8b-serving-cert" not found Jul 29 16:27:18.566427 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.566403 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe54bf4-8094-4c13-8af6-24348f1e568b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:18.578731 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:18.578703 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc99g\" (UniqueName: \"kubernetes.io/projected/bbe54bf4-8094-4c13-8af6-24348f1e568b-kube-api-access-bc99g\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:19.070347 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:19.070313 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:19.072880 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:19.072854 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls\") pod \"maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:19.292325 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:19.292283 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:19.448995 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:19.448957 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f"] Jul 29 16:27:19.452459 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:27:19.452429 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe54bf4_8094_4c13_8af6_24348f1e568b.slice/crio-411b4f5a2b92afd8b5466f4fad1ed0f3eed0736f1010729e88dfcc55ee8c1a96 WatchSource:0}: Error finding container 411b4f5a2b92afd8b5466f4fad1ed0f3eed0736f1010729e88dfcc55ee8c1a96: Status 404 returned error can't find the container with id 411b4f5a2b92afd8b5466f4fad1ed0f3eed0736f1010729e88dfcc55ee8c1a96 Jul 29 16:27:19.489948 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:19.489911 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" event={"ID":"bbe54bf4-8094-4c13-8af6-24348f1e568b","Type":"ContainerStarted","Data":"411b4f5a2b92afd8b5466f4fad1ed0f3eed0736f1010729e88dfcc55ee8c1a96"} Jul 29 16:27:20.496515 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:20.496480 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" event={"ID":"bbe54bf4-8094-4c13-8af6-24348f1e568b","Type":"ContainerStarted","Data":"206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87"} Jul 29 16:27:20.496938 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:20.496543 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:20.517155 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:20.517102 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" podStartSLOduration=2.000762786 podStartE2EDuration="2.517083824s" totalImagesPullingTime="516.321038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:27:20.514871381 +0000 UTC m=+2451.055772851" watchObservedRunningTime="2026-07-29 16:27:20.517083824 +0000 UTC m=+2451.057985293" Jul 29 16:27:26.507757 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:26.507724 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:27:47.678822 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.678784 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv"] Jul 29 16:27:47.682291 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.682274 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.685302 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.685285 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-eccd34-serving-cert\"" Jul 29 16:27:47.692751 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.692717 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv"] Jul 29 16:27:47.732102 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.732059 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91105a3d-2c55-4f81-9038-12b315b952bf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.732283 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.732167 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxfp\" (UniqueName: \"kubernetes.io/projected/91105a3d-2c55-4f81-9038-12b315b952bf-kube-api-access-7xxfp\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.732283 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.732198 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.833721 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.833680 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxfp\" (UniqueName: \"kubernetes.io/projected/91105a3d-2c55-4f81-9038-12b315b952bf-kube-api-access-7xxfp\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.833892 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.833732 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.833892 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.833794 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91105a3d-2c55-4f81-9038-12b315b952bf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.834030 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:27:47.833893 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-serving-cert: secret "maas-api-e2e-shared-b-eccd34-serving-cert" not found Jul 29 16:27:47.834030 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:27:47.833972 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls podName:91105a3d-2c55-4f81-9038-12b315b952bf nodeName:}" failed. No retries permitted until 2026-07-29 16:27:48.333953861 +0000 UTC m=+2478.874855308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls") pod "maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" (UID: "91105a3d-2c55-4f81-9038-12b315b952bf") : secret "maas-api-e2e-shared-b-eccd34-serving-cert" not found Jul 29 16:27:47.834397 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.834381 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91105a3d-2c55-4f81-9038-12b315b952bf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:47.843195 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:47.843166 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxfp\" (UniqueName: \"kubernetes.io/projected/91105a3d-2c55-4f81-9038-12b315b952bf-kube-api-access-7xxfp\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:48.339703 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:48.339666 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:48.342243 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:48.342200 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:48.593972 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:48.593884 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:48.776355 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:48.776327 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv"] Jul 29 16:27:48.778081 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:27:48.778055 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91105a3d_2c55_4f81_9038_12b315b952bf.slice/crio-d2e54362c1b7f109d74b0ac26fc4edf8874d74dc98c5e62cde7d8b7d854d2dfb WatchSource:0}: Error finding container d2e54362c1b7f109d74b0ac26fc4edf8874d74dc98c5e62cde7d8b7d854d2dfb: Status 404 returned error can't find the container with id d2e54362c1b7f109d74b0ac26fc4edf8874d74dc98c5e62cde7d8b7d854d2dfb Jul 29 16:27:49.654653 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:49.654549 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" event={"ID":"91105a3d-2c55-4f81-9038-12b315b952bf","Type":"ContainerStarted","Data":"605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a"} Jul 29 16:27:49.654653 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:49.654633 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" event={"ID":"91105a3d-2c55-4f81-9038-12b315b952bf","Type":"ContainerStarted","Data":"d2e54362c1b7f109d74b0ac26fc4edf8874d74dc98c5e62cde7d8b7d854d2dfb"} Jul 29 16:27:49.654857 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:49.654760 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:27:49.674480 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:49.674428 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" podStartSLOduration=2.143969846 podStartE2EDuration="2.674412388s" totalImagesPullingTime="530.442542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:27:49.672497275 +0000 UTC m=+2480.213398765" watchObservedRunningTime="2026-07-29 16:27:49.674412388 +0000 UTC m=+2480.215313856" Jul 29 16:27:55.666018 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:27:55.665985 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:31:30.096385 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:31:30.096310 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:31:30.108576 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:31:30.108550 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:31:31.102862 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:31:31.102824 2555 scope.go:117] "RemoveContainer" containerID="686119011e90afb17c0f3ee4d92b0b5b062dc54ddaf9002cde5b5fa9e7e0782b" Jul 29 16:33:31.133600 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:31.133556 2555 scope.go:117] "RemoveContainer" containerID="4a8847af58b688bd02ff1ff44ac9b48b1298f861e947dd1198b0ce588404e064" Jul 29 16:33:33.866101 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:33.866061 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl"] Jul 29 16:33:33.870815 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:33.870795 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:33.873656 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:33.873636 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-81ab8b\"/\"openshift-service-ca.crt\"" Jul 29 16:33:33.874812 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:33.874794 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-81ab8b\"/\"default-dockercfg-knk2z\"" Jul 29 16:33:33.874883 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:33.874851 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-81ab8b\"/\"kube-root-ca.crt\"" Jul 29 16:33:33.874927 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:33.874852 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-81ab8b\"/\"sub-test-model-81ab8b-kserve-self-signed-certs\"" Jul 29 16:33:33.886486 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:33.886451 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl"] Jul 29 16:33:34.001522 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.001481 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-tmp-dir\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.001760 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.001531 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/ddc34d86-2587-4106-9108-91051c563f25-kube-api-access-pl4rl\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.001760 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.001637 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-dshm\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.001760 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.001669 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc34d86-2587-4106-9108-91051c563f25-tls-certs\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.001760 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.001717 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-model-cache\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.001932 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.001769 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-home\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.103170 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103110 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-dshm\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.103170 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103155 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc34d86-2587-4106-9108-91051c563f25-tls-certs\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.104498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103205 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-model-cache\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.104498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103252 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-home\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.104498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103317 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-tmp-dir\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.104498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103343 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/ddc34d86-2587-4106-9108-91051c563f25-kube-api-access-pl4rl\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.104498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-model-cache\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.104498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.103984 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-home\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.104498 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.104253 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-tmp-dir\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.106212 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.106190 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-dshm\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.106945 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.106923 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc34d86-2587-4106-9108-91051c563f25-tls-certs\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.112167 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.112140 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/ddc34d86-2587-4106-9108-91051c563f25-kube-api-access-pl4rl\") pod \"sub-test-model-81ab8b-kserve-777bb84b75-g7qsl\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.180793 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.180702 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:34.322119 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.322089 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl"] Jul 29 16:33:34.323749 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:33:34.323721 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddc34d86_2587_4106_9108_91051c563f25.slice/crio-ada3263c9049a4486e46ee84eceddaa4a4d532dff597337a457d51af74ed4ba1 WatchSource:0}: Error finding container ada3263c9049a4486e46ee84eceddaa4a4d532dff597337a457d51af74ed4ba1: Status 404 returned error can't find the container with id ada3263c9049a4486e46ee84eceddaa4a4d532dff597337a457d51af74ed4ba1 Jul 29 16:33:34.457240 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.457156 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" event={"ID":"ddc34d86-2587-4106-9108-91051c563f25","Type":"ContainerStarted","Data":"da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505"} Jul 29 16:33:34.457240 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.457201 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" event={"ID":"ddc34d86-2587-4106-9108-91051c563f25","Type":"ContainerStarted","Data":"ada3263c9049a4486e46ee84eceddaa4a4d532dff597337a457d51af74ed4ba1"} Jul 29 16:33:34.477643 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:34.477567 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" podStartSLOduration=1.477551103 podStartE2EDuration="1.477551103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:33:34.474977184 +0000 UTC m=+2825.015878654" watchObservedRunningTime="2026-07-29 16:33:34.477551103 +0000 UTC m=+2825.018452589" Jul 29 16:33:44.181729 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:44.181688 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:44.181729 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:44.181735 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:44.194497 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:44.194466 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:33:44.523815 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:33:44.523785 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:35:05.734534 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:05.734489 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl"] Jul 29 16:35:20.754015 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:20.753977 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" podUID="ddc34d86-2587-4106-9108-91051c563f25" containerName="main" containerID="cri-o://da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505" gracePeriod=45 Jul 29 16:35:21.009195 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.009127 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:35:21.009690 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.009668 2555 generic.go:356] "Generic (PLEG): container finished" podID="ddc34d86-2587-4106-9108-91051c563f25" containerID="da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505" exitCode=0 Jul 29 16:35:21.009775 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.009737 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" event={"ID":"ddc34d86-2587-4106-9108-91051c563f25","Type":"ContainerDied","Data":"da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505"} Jul 29 16:35:21.009819 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.009774 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" event={"ID":"ddc34d86-2587-4106-9108-91051c563f25","Type":"ContainerDied","Data":"ada3263c9049a4486e46ee84eceddaa4a4d532dff597337a457d51af74ed4ba1"} Jul 29 16:35:21.009819 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.009791 2555 scope.go:117] "RemoveContainer" containerID="da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505" Jul 29 16:35:21.018501 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.018480 2555 scope.go:117] "RemoveContainer" containerID="da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505" Jul 29 16:35:21.018805 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:35:21.018786 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505\": container with ID starting with da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505 not found: ID does not exist" containerID="da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505" Jul 29 16:35:21.018875 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.018813 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505"} err="failed to get container status \"da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505\": rpc error: code = NotFound desc = could not find container \"da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505\": container with ID starting with da7f20a9d28c59ee4e1089c53396053b4664c568d33dfab521f9d0c965795505 not found: ID does not exist" Jul 29 16:35:21.184134 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184098 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc34d86-2587-4106-9108-91051c563f25-tls-certs\") pod \"ddc34d86-2587-4106-9108-91051c563f25\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " Jul 29 16:35:21.184308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184160 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-dshm\") pod \"ddc34d86-2587-4106-9108-91051c563f25\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " Jul 29 16:35:21.184308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184181 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-tmp-dir\") pod \"ddc34d86-2587-4106-9108-91051c563f25\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " Jul 29 16:35:21.184308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184230 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/ddc34d86-2587-4106-9108-91051c563f25-kube-api-access-pl4rl\") pod \"ddc34d86-2587-4106-9108-91051c563f25\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " Jul 29 16:35:21.184308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184277 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-model-cache\") pod \"ddc34d86-2587-4106-9108-91051c563f25\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " Jul 29 16:35:21.184308 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184303 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-home\") pod \"ddc34d86-2587-4106-9108-91051c563f25\" (UID: \"ddc34d86-2587-4106-9108-91051c563f25\") " Jul 29 16:35:21.184542 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184468 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddc34d86-2587-4106-9108-91051c563f25" (UID: "ddc34d86-2587-4106-9108-91051c563f25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:35:21.184671 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184612 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddc34d86-2587-4106-9108-91051c563f25" (UID: "ddc34d86-2587-4106-9108-91051c563f25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:35:21.184671 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184663 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:35:21.184837 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.184677 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-home" (OuterVolumeSpecName: "home") pod "ddc34d86-2587-4106-9108-91051c563f25" (UID: "ddc34d86-2587-4106-9108-91051c563f25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:35:21.186549 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.186522 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-dshm" (OuterVolumeSpecName: "dshm") pod "ddc34d86-2587-4106-9108-91051c563f25" (UID: "ddc34d86-2587-4106-9108-91051c563f25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:35:21.186549 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.186530 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc34d86-2587-4106-9108-91051c563f25-kube-api-access-pl4rl" (OuterVolumeSpecName: "kube-api-access-pl4rl") pod "ddc34d86-2587-4106-9108-91051c563f25" (UID: "ddc34d86-2587-4106-9108-91051c563f25"). InnerVolumeSpecName "kube-api-access-pl4rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:35:21.186718 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.186526 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc34d86-2587-4106-9108-91051c563f25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddc34d86-2587-4106-9108-91051c563f25" (UID: "ddc34d86-2587-4106-9108-91051c563f25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:35:21.285659 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.285621 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:35:21.285659 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.285653 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/ddc34d86-2587-4106-9108-91051c563f25-kube-api-access-pl4rl\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:35:21.285659 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.285663 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:35:21.285908 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.285675 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc34d86-2587-4106-9108-91051c563f25-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:35:21.285908 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:21.285683 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc34d86-2587-4106-9108-91051c563f25-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:35:22.016131 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:22.016095 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl" Jul 29 16:35:22.036286 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:22.036254 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl"] Jul 29 16:35:22.040222 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:22.040194 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-81ab8b/sub-test-model-81ab8b-kserve-777bb84b75-g7qsl"] Jul 29 16:35:23.923936 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:35:23.923901 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc34d86-2587-4106-9108-91051c563f25" path="/var/lib/kubelet/pods/ddc34d86-2587-4106-9108-91051c563f25/volumes" Jul 29 16:36:30.129490 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:36:30.129456 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:36:30.141978 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:36:30.141951 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:37:08.775858 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.775820 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7654-139973292300304"] Jul 29 16:37:08.776369 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.776199 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc34d86-2587-4106-9108-91051c563f25" containerName="main" Jul 29 16:37:08.776369 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.776210 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc34d86-2587-4106-9108-91051c563f25" containerName="main" Jul 29 16:37:08.776369 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.776271 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc34d86-2587-4106-9108-91051c563f25" containerName="main" Jul 29 16:37:08.779390 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.779372 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292300304" Jul 29 16:37:08.782504 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.782480 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lqpzd\"" Jul 29 16:37:08.787718 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.787690 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-139973292300304"] Jul 29 16:37:08.865221 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.865173 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxzw\" (UniqueName: \"kubernetes.io/projected/eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d-kube-api-access-hzxzw\") pod \"test-curl-7654-139973292300304\" (UID: \"eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d\") " pod="opendatahub/test-curl-7654-139973292300304" Jul 29 16:37:08.965693 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.965648 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxzw\" (UniqueName: \"kubernetes.io/projected/eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d-kube-api-access-hzxzw\") pod \"test-curl-7654-139973292300304\" (UID: \"eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d\") " pod="opendatahub/test-curl-7654-139973292300304" Jul 29 16:37:08.973788 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:08.973753 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxzw\" (UniqueName: \"kubernetes.io/projected/eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d-kube-api-access-hzxzw\") pod \"test-curl-7654-139973292300304\" (UID: \"eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d\") " pod="opendatahub/test-curl-7654-139973292300304" Jul 29 16:37:09.090286 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:09.090193 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292300304" Jul 29 16:37:09.438857 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:09.438826 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-139973292300304"] Jul 29 16:37:09.440811 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:37:09.440784 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeef73bb7_2e7f_47b7_80a1_14c9e9a1fa9d.slice/crio-2f75f720f0a9b37765ec6edcf6b925ea81503f55e02f69637783ea466d62f025 WatchSource:0}: Error finding container 2f75f720f0a9b37765ec6edcf6b925ea81503f55e02f69637783ea466d62f025: Status 404 returned error can't find the container with id 2f75f720f0a9b37765ec6edcf6b925ea81503f55e02f69637783ea466d62f025 Jul 29 16:37:09.442194 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:09.442176 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 16:37:09.569025 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:09.568991 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-139973292300304" event={"ID":"eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d","Type":"ContainerStarted","Data":"2f75f720f0a9b37765ec6edcf6b925ea81503f55e02f69637783ea466d62f025"} Jul 29 16:37:10.575999 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:10.575961 2555 generic.go:356] "Generic (PLEG): container finished" podID="eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d" containerID="c0e167feb58e8be015d219eb050111da2cfb31b1dddc3d3d393c21166ab2e04c" exitCode=0 Jul 29 16:37:10.576439 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:10.576051 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-139973292300304" event={"ID":"eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d","Type":"ContainerDied","Data":"c0e167feb58e8be015d219eb050111da2cfb31b1dddc3d3d393c21166ab2e04c"} Jul 29 16:37:11.719917 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.719893 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292300304" Jul 29 16:37:11.757133 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.757101 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7654-139973292300304_eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d/test-curl-7654-139973292300304/0.log" Jul 29 16:37:11.782165 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.782132 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7654-139973292300304"] Jul 29 16:37:11.784314 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.784279 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7654-139973292300304"] Jul 29 16:37:11.792237 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.792197 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzxzw\" (UniqueName: \"kubernetes.io/projected/eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d-kube-api-access-hzxzw\") pod \"eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d\" (UID: \"eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d\") " Jul 29 16:37:11.794340 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.794310 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d-kube-api-access-hzxzw" (OuterVolumeSpecName: "kube-api-access-hzxzw") pod "eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d" (UID: "eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d"). InnerVolumeSpecName "kube-api-access-hzxzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:37:11.893017 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.892918 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxzw\" (UniqueName: \"kubernetes.io/projected/eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d-kube-api-access-hzxzw\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:37:11.925653 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:11.925620 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d" path="/var/lib/kubelet/pods/eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d/volumes" Jul 29 16:37:12.587042 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:12.587012 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292300304" Jul 29 16:37:12.587241 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:12.587062 2555 scope.go:117] "RemoveContainer" containerID="c0e167feb58e8be015d219eb050111da2cfb31b1dddc3d3d393c21166ab2e04c" Jul 29 16:37:21.778368 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.778277 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f"] Jul 29 16:37:21.778832 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.778667 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d" containerName="test-curl-7654-139973292300304" Jul 29 16:37:21.778832 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.778679 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d" containerName="test-curl-7654-139973292300304" Jul 29 16:37:21.778832 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.778744 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef73bb7-2e7f-47b7-80a1-14c9e9a1fa9d" containerName="test-curl-7654-139973292300304" Jul 29 16:37:21.783199 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.783177 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.786069 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.786046 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-777250-serving-cert\"" Jul 29 16:37:21.794733 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.794707 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f"] Jul 29 16:37:21.882827 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.882790 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.883019 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.882850 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhm7s\" (UniqueName: \"kubernetes.io/projected/40ae9a3c-7184-47ec-a43b-f162d15be3b6-kube-api-access-qhm7s\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.883019 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.882919 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ae9a3c-7184-47ec-a43b-f162d15be3b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.983652 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.983613 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.983839 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.983677 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhm7s\" (UniqueName: \"kubernetes.io/projected/40ae9a3c-7184-47ec-a43b-f162d15be3b6-kube-api-access-qhm7s\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.983839 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.983705 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ae9a3c-7184-47ec-a43b-f162d15be3b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.983839 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:37:21.983758 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-serving-cert: secret "maas-api-e2e-ipp-a-777250-serving-cert" not found Jul 29 16:37:21.983839 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:37:21.983834 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls podName:40ae9a3c-7184-47ec-a43b-f162d15be3b6 nodeName:}" failed. No retries permitted until 2026-07-29 16:37:22.483816138 +0000 UTC m=+3053.024717603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls") pod "maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" (UID: "40ae9a3c-7184-47ec-a43b-f162d15be3b6") : secret "maas-api-e2e-ipp-a-777250-serving-cert" not found Jul 29 16:37:21.984447 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.984429 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ae9a3c-7184-47ec-a43b-f162d15be3b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:21.996472 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:21.996435 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhm7s\" (UniqueName: \"kubernetes.io/projected/40ae9a3c-7184-47ec-a43b-f162d15be3b6-kube-api-access-qhm7s\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:22.487903 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:22.487863 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:22.490393 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:22.490374 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:22.694210 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:22.694173 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:22.840633 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:22.840608 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f"] Jul 29 16:37:22.842796 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:37:22.842769 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ae9a3c_7184_47ec_a43b_f162d15be3b6.slice/crio-ca2ac883f852ed14cdc42a3729d366ee5d321d2ce63ebaa08cfd94a82ac41bc2 WatchSource:0}: Error finding container ca2ac883f852ed14cdc42a3729d366ee5d321d2ce63ebaa08cfd94a82ac41bc2: Status 404 returned error can't find the container with id ca2ac883f852ed14cdc42a3729d366ee5d321d2ce63ebaa08cfd94a82ac41bc2 Jul 29 16:37:23.652740 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:23.652648 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" event={"ID":"40ae9a3c-7184-47ec-a43b-f162d15be3b6","Type":"ContainerStarted","Data":"a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56"} Jul 29 16:37:23.652740 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:23.652705 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" event={"ID":"40ae9a3c-7184-47ec-a43b-f162d15be3b6","Type":"ContainerStarted","Data":"ca2ac883f852ed14cdc42a3729d366ee5d321d2ce63ebaa08cfd94a82ac41bc2"} Jul 29 16:37:23.652976 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:23.652856 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:23.671658 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:23.671604 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" podStartSLOduration=2.177482775 podStartE2EDuration="2.671566524s" totalImagesPullingTime="494.083749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:37:23.669911476 +0000 UTC m=+3054.210812945" watchObservedRunningTime="2026-07-29 16:37:23.671566524 +0000 UTC m=+3054.212467992" Jul 29 16:37:29.663845 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:29.663816 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:37:50.764800 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:50.764764 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv"] Jul 29 16:37:50.770665 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:50.770640 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:50.774051 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:50.774017 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3c2006-serving-cert\"" Jul 29 16:37:50.791824 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:50.791793 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv"] Jul 29 16:37:50.947639 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:50.947573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:50.947831 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:50.947649 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcpmn\" (UniqueName: \"kubernetes.io/projected/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-kube-api-access-tcpmn\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:50.947831 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:50.947714 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.048841 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.048748 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.048841 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.048827 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.048841 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.048847 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcpmn\" (UniqueName: \"kubernetes.io/projected/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-kube-api-access-tcpmn\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.049539 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.049512 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.051447 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.051407 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.058412 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.058380 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcpmn\" (UniqueName: \"kubernetes.io/projected/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-kube-api-access-tcpmn\") pod \"maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.084178 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.084139 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.228854 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.228820 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv"] Jul 29 16:37:51.229987 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:37:51.229949 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc84d5d2_603b_41df_b29f_56fcea7d5dfe.slice/crio-63e37fc0192c1bf613cb1b02c920558566810068ea072d84fe75ab1645dc7a9b WatchSource:0}: Error finding container 63e37fc0192c1bf613cb1b02c920558566810068ea072d84fe75ab1645dc7a9b: Status 404 returned error can't find the container with id 63e37fc0192c1bf613cb1b02c920558566810068ea072d84fe75ab1645dc7a9b Jul 29 16:37:51.802403 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.802364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" event={"ID":"cc84d5d2-603b-41df-b29f-56fcea7d5dfe","Type":"ContainerStarted","Data":"d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b"} Jul 29 16:37:51.802913 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.802430 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:37:51.802913 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.802452 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" event={"ID":"cc84d5d2-603b-41df-b29f-56fcea7d5dfe","Type":"ContainerStarted","Data":"63e37fc0192c1bf613cb1b02c920558566810068ea072d84fe75ab1645dc7a9b"} Jul 29 16:37:51.821700 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:51.821646 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" podStartSLOduration=1.426547275 podStartE2EDuration="1.821629357s" totalImagesPullingTime="395.082082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:37:51.820543754 +0000 UTC m=+3082.361445223" watchObservedRunningTime="2026-07-29 16:37:51.821629357 +0000 UTC m=+3082.362530828" Jul 29 16:37:57.814478 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:37:57.814441 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:38:18.976263 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:18.976225 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99b8z"] Jul 29 16:38:18.979853 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:18.979834 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:18.982764 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:18.982742 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8rmrp\"" Jul 29 16:38:18.982764 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:18.982754 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 16:38:18.982915 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:18.982760 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 16:38:18.989231 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:18.989205 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99b8z"] Jul 29 16:38:19.109746 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:19.109705 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgfn\" (UniqueName: \"kubernetes.io/projected/dab45d29-99e3-4d80-80cc-810fd6ca5539-kube-api-access-xtgfn\") pod \"kuadrant-operator-catalog-99b8z\" (UID: \"dab45d29-99e3-4d80-80cc-810fd6ca5539\") " pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:19.210845 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:19.210805 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgfn\" (UniqueName: \"kubernetes.io/projected/dab45d29-99e3-4d80-80cc-810fd6ca5539-kube-api-access-xtgfn\") pod \"kuadrant-operator-catalog-99b8z\" (UID: \"dab45d29-99e3-4d80-80cc-810fd6ca5539\") " pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:19.219979 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:19.219944 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgfn\" (UniqueName: \"kubernetes.io/projected/dab45d29-99e3-4d80-80cc-810fd6ca5539-kube-api-access-xtgfn\") pod \"kuadrant-operator-catalog-99b8z\" (UID: \"dab45d29-99e3-4d80-80cc-810fd6ca5539\") " pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:19.291750 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:19.291703 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:19.430117 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:19.430084 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99b8z"] Jul 29 16:38:19.434887 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:38:19.434855 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab45d29_99e3_4d80_80cc_810fd6ca5539.slice/crio-882b707f2a1f967d6236d97df204922950f81c901861c2528473475361d6aff4 WatchSource:0}: Error finding container 882b707f2a1f967d6236d97df204922950f81c901861c2528473475361d6aff4: Status 404 returned error can't find the container with id 882b707f2a1f967d6236d97df204922950f81c901861c2528473475361d6aff4 Jul 29 16:38:19.950021 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:19.949985 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" event={"ID":"dab45d29-99e3-4d80-80cc-810fd6ca5539","Type":"ContainerStarted","Data":"882b707f2a1f967d6236d97df204922950f81c901861c2528473475361d6aff4"} Jul 29 16:38:21.962908 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:21.962871 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" event={"ID":"dab45d29-99e3-4d80-80cc-810fd6ca5539","Type":"ContainerStarted","Data":"0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e"} Jul 29 16:38:21.981938 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:21.981882 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" podStartSLOduration=2.21378735 podStartE2EDuration="3.981866897s" totalImagesPullingTime="1.768079547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:38:21.980161716 +0000 UTC m=+3112.521063185" watchObservedRunningTime="2026-07-29 16:38:21.981866897 +0000 UTC m=+3112.522768366" Jul 29 16:38:27.736718 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:27.736685 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hpvlw_cc9cbd65-0255-4ed1-83ad-3d534ec9d42d/payload-processing/0.log" Jul 29 16:38:29.292699 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:29.292659 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:29.293226 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:29.292723 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:29.315055 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:29.315017 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:30.029527 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:30.029498 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:30.065972 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:30.065940 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99b8z"] Jul 29 16:38:32.017782 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:32.017741 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" podUID="dab45d29-99e3-4d80-80cc-810fd6ca5539" containerName="registry-server" containerID="cri-o://0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e" gracePeriod=2 Jul 29 16:38:32.275377 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:32.275312 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:32.433015 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:32.432976 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtgfn\" (UniqueName: \"kubernetes.io/projected/dab45d29-99e3-4d80-80cc-810fd6ca5539-kube-api-access-xtgfn\") pod \"dab45d29-99e3-4d80-80cc-810fd6ca5539\" (UID: \"dab45d29-99e3-4d80-80cc-810fd6ca5539\") " Jul 29 16:38:32.435082 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:32.435047 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab45d29-99e3-4d80-80cc-810fd6ca5539-kube-api-access-xtgfn" (OuterVolumeSpecName: "kube-api-access-xtgfn") pod "dab45d29-99e3-4d80-80cc-810fd6ca5539" (UID: "dab45d29-99e3-4d80-80cc-810fd6ca5539"). InnerVolumeSpecName "kube-api-access-xtgfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:38:32.534184 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:32.534096 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtgfn\" (UniqueName: \"kubernetes.io/projected/dab45d29-99e3-4d80-80cc-810fd6ca5539-kube-api-access-xtgfn\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:38:33.024857 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.024816 2555 generic.go:356] "Generic (PLEG): container finished" podID="dab45d29-99e3-4d80-80cc-810fd6ca5539" containerID="0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e" exitCode=0 Jul 29 16:38:33.025387 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.024880 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" Jul 29 16:38:33.025387 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.024902 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" event={"ID":"dab45d29-99e3-4d80-80cc-810fd6ca5539","Type":"ContainerDied","Data":"0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e"} Jul 29 16:38:33.025387 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.024977 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99b8z" event={"ID":"dab45d29-99e3-4d80-80cc-810fd6ca5539","Type":"ContainerDied","Data":"882b707f2a1f967d6236d97df204922950f81c901861c2528473475361d6aff4"} Jul 29 16:38:33.025387 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.025010 2555 scope.go:117] "RemoveContainer" containerID="0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e" Jul 29 16:38:33.034886 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.034867 2555 scope.go:117] "RemoveContainer" containerID="0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e" Jul 29 16:38:33.035145 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:38:33.035124 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e\": container with ID starting with 0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e not found: ID does not exist" containerID="0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e" Jul 29 16:38:33.035208 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.035152 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e"} err="failed to get container status \"0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e\": rpc error: code = NotFound desc = could not find container \"0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e\": container with ID starting with 0b5d1cbc22d4f8c1cbd39386d371916facdea8ca012cc086ff1c04adce47be2e not found: ID does not exist" Jul 29 16:38:33.048026 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.047987 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99b8z"] Jul 29 16:38:33.051892 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.051866 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99b8z"] Jul 29 16:38:33.926567 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:38:33.926532 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab45d29-99e3-4d80-80cc-810fd6ca5539" path="/var/lib/kubelet/pods/dab45d29-99e3-4d80-80cc-810fd6ca5539/volumes" Jul 29 16:39:14.949849 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:14.949815 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hpvlw_cc9cbd65-0255-4ed1-83ad-3d534ec9d42d/payload-processing/0.log" Jul 29 16:39:19.211447 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.211409 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5"] Jul 29 16:39:19.211846 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.211832 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab45d29-99e3-4d80-80cc-810fd6ca5539" containerName="registry-server" Jul 29 16:39:19.211892 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.211848 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab45d29-99e3-4d80-80cc-810fd6ca5539" containerName="registry-server" Jul 29 16:39:19.211940 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.211930 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab45d29-99e3-4d80-80cc-810fd6ca5539" containerName="registry-server" Jul 29 16:39:19.215058 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.215037 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.217896 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.217875 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1bad22-serving-cert\"" Jul 29 16:39:19.232882 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.232853 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5"] Jul 29 16:39:19.364072 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.364033 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d5e319-6bc6-4952-a291-761d5021414f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.364253 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.364090 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmzh\" (UniqueName: \"kubernetes.io/projected/19d5e319-6bc6-4952-a291-761d5021414f-kube-api-access-dkmzh\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.364253 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.364119 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d5e319-6bc6-4952-a291-761d5021414f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.464974 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.464881 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d5e319-6bc6-4952-a291-761d5021414f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.464974 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.464940 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmzh\" (UniqueName: \"kubernetes.io/projected/19d5e319-6bc6-4952-a291-761d5021414f-kube-api-access-dkmzh\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.465176 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.465059 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d5e319-6bc6-4952-a291-761d5021414f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.465683 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.465663 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d5e319-6bc6-4952-a291-761d5021414f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.467623 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.467601 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d5e319-6bc6-4952-a291-761d5021414f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.474007 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.473983 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmzh\" (UniqueName: \"kubernetes.io/projected/19d5e319-6bc6-4952-a291-761d5021414f-kube-api-access-dkmzh\") pod \"maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.525820 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.525781 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:19.668765 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:19.668737 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5"] Jul 29 16:39:19.670950 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:39:19.670926 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d5e319_6bc6_4952_a291_761d5021414f.slice/crio-d77c5610b08ce7bd0acb80e04d58fa86ec169f22cd976c281c53f1c684a344b0 WatchSource:0}: Error finding container d77c5610b08ce7bd0acb80e04d58fa86ec169f22cd976c281c53f1c684a344b0: Status 404 returned error can't find the container with id d77c5610b08ce7bd0acb80e04d58fa86ec169f22cd976c281c53f1c684a344b0 Jul 29 16:39:20.278426 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:20.278387 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" event={"ID":"19d5e319-6bc6-4952-a291-761d5021414f","Type":"ContainerStarted","Data":"f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782"} Jul 29 16:39:20.278817 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:20.278458 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:20.278817 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:20.278484 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" event={"ID":"19d5e319-6bc6-4952-a291-761d5021414f","Type":"ContainerStarted","Data":"d77c5610b08ce7bd0acb80e04d58fa86ec169f22cd976c281c53f1c684a344b0"} Jul 29 16:39:26.290550 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:26.290517 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:26.310876 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:26.310811 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" podStartSLOduration=6.777592986 podStartE2EDuration="7.310788438s" totalImagesPullingTime="533.195452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:39:20.300515792 +0000 UTC m=+3170.841417260" watchObservedRunningTime="2026-07-29 16:39:26.310788438 +0000 UTC m=+3176.851689915" Jul 29 16:39:44.855613 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:44.855545 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs"] Jul 29 16:39:44.859538 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:44.859508 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" Jul 29 16:39:44.862142 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:44.862118 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:39:44.867521 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:44.866801 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs"] Jul 29 16:39:45.018533 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:45.018499 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b679ad8-61f3-4ea5-93dd-68935b5c972b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs\" (UID: \"1b679ad8-61f3-4ea5-93dd-68935b5c972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" Jul 29 16:39:45.119569 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:45.119460 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b679ad8-61f3-4ea5-93dd-68935b5c972b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs\" (UID: \"1b679ad8-61f3-4ea5-93dd-68935b5c972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" Jul 29 16:39:45.120138 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:45.120118 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b679ad8-61f3-4ea5-93dd-68935b5c972b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs\" (UID: \"1b679ad8-61f3-4ea5-93dd-68935b5c972b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" Jul 29 16:39:45.173478 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:45.173443 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" Jul 29 16:39:45.317743 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:45.317697 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs"] Jul 29 16:39:45.320707 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:39:45.320677 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b679ad8_61f3_4ea5_93dd_68935b5c972b.slice/crio-1421af289fdd3dbd58708a6108198d53418d1c08e018b339b0dcaf30f1663efd WatchSource:0}: Error finding container 1421af289fdd3dbd58708a6108198d53418d1c08e018b339b0dcaf30f1663efd: Status 404 returned error can't find the container with id 1421af289fdd3dbd58708a6108198d53418d1c08e018b339b0dcaf30f1663efd Jul 29 16:39:45.414994 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:45.414910 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" event={"ID":"1b679ad8-61f3-4ea5-93dd-68935b5c972b","Type":"ContainerStarted","Data":"1421af289fdd3dbd58708a6108198d53418d1c08e018b339b0dcaf30f1663efd"} Jul 29 16:39:46.421817 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:46.421782 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b679ad8-61f3-4ea5-93dd-68935b5c972b" containerID="7f2e4cee36f7ad81478136eb10d39236495adf10aba94c87dee154b80898d42e" exitCode=0 Jul 29 16:39:46.422258 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:46.421866 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" event={"ID":"1b679ad8-61f3-4ea5-93dd-68935b5c972b","Type":"ContainerDied","Data":"7f2e4cee36f7ad81478136eb10d39236495adf10aba94c87dee154b80898d42e"} Jul 29 16:39:47.569863 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:47.569836 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" Jul 29 16:39:47.644358 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:47.644300 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b679ad8-61f3-4ea5-93dd-68935b5c972b-maas-api-service-ca\") pod \"1b679ad8-61f3-4ea5-93dd-68935b5c972b\" (UID: \"1b679ad8-61f3-4ea5-93dd-68935b5c972b\") " Jul 29 16:39:47.644685 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:47.644663 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b679ad8-61f3-4ea5-93dd-68935b5c972b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b679ad8-61f3-4ea5-93dd-68935b5c972b" (UID: "1b679ad8-61f3-4ea5-93dd-68935b5c972b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:39:47.746210 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:47.746113 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b679ad8-61f3-4ea5-93dd-68935b5c972b-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:39:48.433542 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:48.433510 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" Jul 29 16:39:48.433740 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:48.433541 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs" event={"ID":"1b679ad8-61f3-4ea5-93dd-68935b5c972b","Type":"ContainerDied","Data":"1421af289fdd3dbd58708a6108198d53418d1c08e018b339b0dcaf30f1663efd"} Jul 29 16:39:48.433740 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:48.433616 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1421af289fdd3dbd58708a6108198d53418d1c08e018b339b0dcaf30f1663efd" Jul 29 16:39:49.830866 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:49.830830 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5"] Jul 29 16:39:49.831366 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:49.831069 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" podUID="19d5e319-6bc6-4952-a291-761d5021414f" containerName="maas-api" containerID="cri-o://f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782" gracePeriod=30 Jul 29 16:39:50.058896 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.058861 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs"] Jul 29 16:39:50.062658 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.062622 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1bad22-98gvs"] Jul 29 16:39:50.095775 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.095710 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:50.268830 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.268796 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d5e319-6bc6-4952-a291-761d5021414f-maas-api-tls\") pod \"19d5e319-6bc6-4952-a291-761d5021414f\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " Jul 29 16:39:50.268830 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.268832 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d5e319-6bc6-4952-a291-761d5021414f-openshift-service-ca\") pod \"19d5e319-6bc6-4952-a291-761d5021414f\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " Jul 29 16:39:50.269056 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.268872 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkmzh\" (UniqueName: \"kubernetes.io/projected/19d5e319-6bc6-4952-a291-761d5021414f-kube-api-access-dkmzh\") pod \"19d5e319-6bc6-4952-a291-761d5021414f\" (UID: \"19d5e319-6bc6-4952-a291-761d5021414f\") " Jul 29 16:39:50.269212 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.269190 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d5e319-6bc6-4952-a291-761d5021414f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19d5e319-6bc6-4952-a291-761d5021414f" (UID: "19d5e319-6bc6-4952-a291-761d5021414f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:39:50.271002 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.270978 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d5e319-6bc6-4952-a291-761d5021414f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19d5e319-6bc6-4952-a291-761d5021414f" (UID: "19d5e319-6bc6-4952-a291-761d5021414f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:39:50.271210 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.271191 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d5e319-6bc6-4952-a291-761d5021414f-kube-api-access-dkmzh" (OuterVolumeSpecName: "kube-api-access-dkmzh") pod "19d5e319-6bc6-4952-a291-761d5021414f" (UID: "19d5e319-6bc6-4952-a291-761d5021414f"). InnerVolumeSpecName "kube-api-access-dkmzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:39:50.369969 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.369879 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d5e319-6bc6-4952-a291-761d5021414f-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:39:50.369969 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.369911 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d5e319-6bc6-4952-a291-761d5021414f-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:39:50.369969 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.369922 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkmzh\" (UniqueName: \"kubernetes.io/projected/19d5e319-6bc6-4952-a291-761d5021414f-kube-api-access-dkmzh\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:39:50.445295 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.445258 2555 generic.go:356] "Generic (PLEG): container finished" podID="19d5e319-6bc6-4952-a291-761d5021414f" containerID="f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782" exitCode=0 Jul 29 16:39:50.445474 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.445308 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" Jul 29 16:39:50.445474 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.445348 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" event={"ID":"19d5e319-6bc6-4952-a291-761d5021414f","Type":"ContainerDied","Data":"f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782"} Jul 29 16:39:50.445474 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.445414 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5" event={"ID":"19d5e319-6bc6-4952-a291-761d5021414f","Type":"ContainerDied","Data":"d77c5610b08ce7bd0acb80e04d58fa86ec169f22cd976c281c53f1c684a344b0"} Jul 29 16:39:50.445474 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.445438 2555 scope.go:117] "RemoveContainer" containerID="f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782" Jul 29 16:39:50.455507 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.455488 2555 scope.go:117] "RemoveContainer" containerID="f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782" Jul 29 16:39:50.455837 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:39:50.455818 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782\": container with ID starting with f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782 not found: ID does not exist" containerID="f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782" Jul 29 16:39:50.455912 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.455847 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782"} err="failed to get container status \"f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782\": rpc error: code = NotFound desc = could not find container \"f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782\": container with ID starting with f25fc49827a909aefc6a2ca69a7dc0ad4452cfb7ff66c62f726f29be9fa6b782 not found: ID does not exist" Jul 29 16:39:50.468909 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.468872 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5"] Jul 29 16:39:50.474733 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:50.474704 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1bad22-7dbc5bc9d7-fzqn5"] Jul 29 16:39:51.924560 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:51.924526 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d5e319-6bc6-4952-a291-761d5021414f" path="/var/lib/kubelet/pods/19d5e319-6bc6-4952-a291-761d5021414f/volumes" Jul 29 16:39:51.924967 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:39:51.924888 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b679ad8-61f3-4ea5-93dd-68935b5c972b" path="/var/lib/kubelet/pods/1b679ad8-61f3-4ea5-93dd-68935b5c972b/volumes" Jul 29 16:40:00.096172 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.096133 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5"] Jul 29 16:40:00.096566 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.096531 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d5e319-6bc6-4952-a291-761d5021414f" containerName="maas-api" Jul 29 16:40:00.096566 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.096544 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d5e319-6bc6-4952-a291-761d5021414f" containerName="maas-api" Jul 29 16:40:00.096566 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.096561 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b679ad8-61f3-4ea5-93dd-68935b5c972b" containerName="revoke-keys" Jul 29 16:40:00.096566 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.096566 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b679ad8-61f3-4ea5-93dd-68935b5c972b" containerName="revoke-keys" Jul 29 16:40:00.096724 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.096658 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d5e319-6bc6-4952-a291-761d5021414f" containerName="maas-api" Jul 29 16:40:00.096724 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.096670 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b679ad8-61f3-4ea5-93dd-68935b5c972b" containerName="revoke-keys" Jul 29 16:40:00.101094 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.101074 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" Jul 29 16:40:00.103675 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.103653 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:40:00.107841 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.107810 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5"] Jul 29 16:40:00.162338 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.162301 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f54e0f-11b6-409b-aac3-fa9690973e38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5\" (UID: \"a4f54e0f-11b6-409b-aac3-fa9690973e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" Jul 29 16:40:00.263684 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.263630 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f54e0f-11b6-409b-aac3-fa9690973e38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5\" (UID: \"a4f54e0f-11b6-409b-aac3-fa9690973e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" Jul 29 16:40:00.264230 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.264207 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f54e0f-11b6-409b-aac3-fa9690973e38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5\" (UID: \"a4f54e0f-11b6-409b-aac3-fa9690973e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" Jul 29 16:40:00.413544 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.413451 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" Jul 29 16:40:00.553037 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:00.553007 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5"] Jul 29 16:40:00.554751 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:40:00.554724 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f54e0f_11b6_409b_aac3_fa9690973e38.slice/crio-71faf6dc9358cf11635f530c9791d0978295d90f378914eb04d79be2456daa30 WatchSource:0}: Error finding container 71faf6dc9358cf11635f530c9791d0978295d90f378914eb04d79be2456daa30: Status 404 returned error can't find the container with id 71faf6dc9358cf11635f530c9791d0978295d90f378914eb04d79be2456daa30 Jul 29 16:40:01.508162 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:01.508128 2555 generic.go:356] "Generic (PLEG): container finished" podID="a4f54e0f-11b6-409b-aac3-fa9690973e38" containerID="279be1230f99fd3aa77f45d97fc6cf67ddb579905687b16036bde1b71233dbb0" exitCode=0 Jul 29 16:40:01.508533 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:01.508206 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" event={"ID":"a4f54e0f-11b6-409b-aac3-fa9690973e38","Type":"ContainerDied","Data":"279be1230f99fd3aa77f45d97fc6cf67ddb579905687b16036bde1b71233dbb0"} Jul 29 16:40:01.508533 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:01.508257 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" event={"ID":"a4f54e0f-11b6-409b-aac3-fa9690973e38","Type":"ContainerStarted","Data":"71faf6dc9358cf11635f530c9791d0978295d90f378914eb04d79be2456daa30"} Jul 29 16:40:02.648857 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:02.648824 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" Jul 29 16:40:02.686879 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:02.686833 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f54e0f-11b6-409b-aac3-fa9690973e38-maas-api-service-ca\") pod \"a4f54e0f-11b6-409b-aac3-fa9690973e38\" (UID: \"a4f54e0f-11b6-409b-aac3-fa9690973e38\") " Jul 29 16:40:02.687214 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:02.687186 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f54e0f-11b6-409b-aac3-fa9690973e38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4f54e0f-11b6-409b-aac3-fa9690973e38" (UID: "a4f54e0f-11b6-409b-aac3-fa9690973e38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:40:02.788641 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:02.788517 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f54e0f-11b6-409b-aac3-fa9690973e38-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:40:03.520348 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:03.520318 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" Jul 29 16:40:03.520348 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:03.520334 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5" event={"ID":"a4f54e0f-11b6-409b-aac3-fa9690973e38","Type":"ContainerDied","Data":"71faf6dc9358cf11635f530c9791d0978295d90f378914eb04d79be2456daa30"} Jul 29 16:40:03.520574 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:03.520383 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71faf6dc9358cf11635f530c9791d0978295d90f378914eb04d79be2456daa30" Jul 29 16:40:15.699362 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:15.699330 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f"] Jul 29 16:40:15.699795 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:15.699527 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" podUID="40ae9a3c-7184-47ec-a43b-f162d15be3b6" containerName="maas-api" containerID="cri-o://a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56" gracePeriod=30 Jul 29 16:40:15.928556 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:15.928523 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5"] Jul 29 16:40:15.928556 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:15.928562 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-777250-xrzc5"] Jul 29 16:40:15.958091 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:15.958025 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:40:16.014439 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.014401 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhm7s\" (UniqueName: \"kubernetes.io/projected/40ae9a3c-7184-47ec-a43b-f162d15be3b6-kube-api-access-qhm7s\") pod \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " Jul 29 16:40:16.014666 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.014455 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ae9a3c-7184-47ec-a43b-f162d15be3b6-openshift-service-ca\") pod \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " Jul 29 16:40:16.014666 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.014547 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls\") pod \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\" (UID: \"40ae9a3c-7184-47ec-a43b-f162d15be3b6\") " Jul 29 16:40:16.014867 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.014843 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ae9a3c-7184-47ec-a43b-f162d15be3b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40ae9a3c-7184-47ec-a43b-f162d15be3b6" (UID: "40ae9a3c-7184-47ec-a43b-f162d15be3b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:40:16.014951 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.014939 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ae9a3c-7184-47ec-a43b-f162d15be3b6-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:40:16.016616 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.016572 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40ae9a3c-7184-47ec-a43b-f162d15be3b6" (UID: "40ae9a3c-7184-47ec-a43b-f162d15be3b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:40:16.016616 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.016594 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ae9a3c-7184-47ec-a43b-f162d15be3b6-kube-api-access-qhm7s" (OuterVolumeSpecName: "kube-api-access-qhm7s") pod "40ae9a3c-7184-47ec-a43b-f162d15be3b6" (UID: "40ae9a3c-7184-47ec-a43b-f162d15be3b6"). InnerVolumeSpecName "kube-api-access-qhm7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:40:16.116379 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.116346 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40ae9a3c-7184-47ec-a43b-f162d15be3b6-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:40:16.116379 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.116375 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhm7s\" (UniqueName: \"kubernetes.io/projected/40ae9a3c-7184-47ec-a43b-f162d15be3b6-kube-api-access-qhm7s\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:40:16.588702 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.588670 2555 generic.go:356] "Generic (PLEG): container finished" podID="40ae9a3c-7184-47ec-a43b-f162d15be3b6" containerID="a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56" exitCode=0 Jul 29 16:40:16.588977 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.588718 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" Jul 29 16:40:16.588977 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.588737 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" event={"ID":"40ae9a3c-7184-47ec-a43b-f162d15be3b6","Type":"ContainerDied","Data":"a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56"} Jul 29 16:40:16.588977 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.588779 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f" event={"ID":"40ae9a3c-7184-47ec-a43b-f162d15be3b6","Type":"ContainerDied","Data":"ca2ac883f852ed14cdc42a3729d366ee5d321d2ce63ebaa08cfd94a82ac41bc2"} Jul 29 16:40:16.588977 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.588809 2555 scope.go:117] "RemoveContainer" containerID="a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56" Jul 29 16:40:16.601156 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.600307 2555 scope.go:117] "RemoveContainer" containerID="a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56" Jul 29 16:40:16.601415 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:40:16.601247 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56\": container with ID starting with a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56 not found: ID does not exist" containerID="a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56" Jul 29 16:40:16.601415 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.601287 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56"} err="failed to get container status \"a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56\": rpc error: code = NotFound desc = could not find container \"a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56\": container with ID starting with a717a944df36eafb61a78726c43eba2b96d72ecca6950779acd707fd5ec9fd56 not found: ID does not exist" Jul 29 16:40:16.617299 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.617267 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f"] Jul 29 16:40:16.622664 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:16.622634 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-777250-5fb4c94df4-mvp9f"] Jul 29 16:40:17.924016 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:17.923984 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ae9a3c-7184-47ec-a43b-f162d15be3b6" path="/var/lib/kubelet/pods/40ae9a3c-7184-47ec-a43b-f162d15be3b6/volumes" Jul 29 16:40:17.924520 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:40:17.924482 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f54e0f-11b6-409b-aac3-fa9690973e38" path="/var/lib/kubelet/pods/a4f54e0f-11b6-409b-aac3-fa9690973e38/volumes" Jul 29 16:41:04.922686 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.922644 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb"] Jul 29 16:41:04.923178 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.923062 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ae9a3c-7184-47ec-a43b-f162d15be3b6" containerName="maas-api" Jul 29 16:41:04.923178 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.923075 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ae9a3c-7184-47ec-a43b-f162d15be3b6" containerName="maas-api" Jul 29 16:41:04.923178 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.923083 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f54e0f-11b6-409b-aac3-fa9690973e38" containerName="revoke-keys" Jul 29 16:41:04.923178 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.923089 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f54e0f-11b6-409b-aac3-fa9690973e38" containerName="revoke-keys" Jul 29 16:41:04.923178 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.923149 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ae9a3c-7184-47ec-a43b-f162d15be3b6" containerName="maas-api" Jul 29 16:41:04.923178 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.923159 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f54e0f-11b6-409b-aac3-fa9690973e38" containerName="revoke-keys" Jul 29 16:41:04.926431 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.926409 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" Jul 29 16:41:04.929386 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.929359 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:41:04.933971 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:04.933945 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb"] Jul 29 16:41:05.084649 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.084573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9025c1-d85d-462d-9e62-669541df0845-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb\" (UID: \"0d9025c1-d85d-462d-9e62-669541df0845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" Jul 29 16:41:05.185439 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.185338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9025c1-d85d-462d-9e62-669541df0845-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb\" (UID: \"0d9025c1-d85d-462d-9e62-669541df0845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" Jul 29 16:41:05.186026 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.186007 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9025c1-d85d-462d-9e62-669541df0845-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb\" (UID: \"0d9025c1-d85d-462d-9e62-669541df0845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" Jul 29 16:41:05.239400 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.239357 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" Jul 29 16:41:05.375849 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.375818 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb"] Jul 29 16:41:05.377727 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:41:05.377699 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9025c1_d85d_462d_9e62_669541df0845.slice/crio-3ae4dabaf1843252502eb8a5a8016176c104debc759742aa5189983a9ec02b54 WatchSource:0}: Error finding container 3ae4dabaf1843252502eb8a5a8016176c104debc759742aa5189983a9ec02b54: Status 404 returned error can't find the container with id 3ae4dabaf1843252502eb8a5a8016176c104debc759742aa5189983a9ec02b54 Jul 29 16:41:05.860607 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.860502 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" event={"ID":"0d9025c1-d85d-462d-9e62-669541df0845","Type":"ContainerStarted","Data":"193e2ca92f21da0b3a48f9ea1db619cc68557f96095aed713e87ba4509b7f29b"} Jul 29 16:41:05.860607 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.860557 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" event={"ID":"0d9025c1-d85d-462d-9e62-669541df0845","Type":"ContainerStarted","Data":"3ae4dabaf1843252502eb8a5a8016176c104debc759742aa5189983a9ec02b54"} Jul 29 16:41:05.879341 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:05.879290 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" podStartSLOduration=1.879273953 podStartE2EDuration="1.879273953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:41:05.876762793 +0000 UTC m=+3276.417664273" watchObservedRunningTime="2026-07-29 16:41:05.879273953 +0000 UTC m=+3276.420175418" Jul 29 16:41:06.867792 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:06.867760 2555 generic.go:356] "Generic (PLEG): container finished" podID="0d9025c1-d85d-462d-9e62-669541df0845" containerID="193e2ca92f21da0b3a48f9ea1db619cc68557f96095aed713e87ba4509b7f29b" exitCode=0 Jul 29 16:41:06.868185 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:06.867812 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" event={"ID":"0d9025c1-d85d-462d-9e62-669541df0845","Type":"ContainerDied","Data":"193e2ca92f21da0b3a48f9ea1db619cc68557f96095aed713e87ba4509b7f29b"} Jul 29 16:41:08.006182 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:08.006158 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" Jul 29 16:41:08.116693 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:08.116658 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9025c1-d85d-462d-9e62-669541df0845-maas-api-service-ca\") pod \"0d9025c1-d85d-462d-9e62-669541df0845\" (UID: \"0d9025c1-d85d-462d-9e62-669541df0845\") " Jul 29 16:41:08.117058 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:08.117037 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9025c1-d85d-462d-9e62-669541df0845-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d9025c1-d85d-462d-9e62-669541df0845" (UID: "0d9025c1-d85d-462d-9e62-669541df0845"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:08.217798 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:08.217712 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9025c1-d85d-462d-9e62-669541df0845-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:08.879345 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:08.879309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" event={"ID":"0d9025c1-d85d-462d-9e62-669541df0845","Type":"ContainerDied","Data":"3ae4dabaf1843252502eb8a5a8016176c104debc759742aa5189983a9ec02b54"} Jul 29 16:41:08.879524 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:08.879359 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae4dabaf1843252502eb8a5a8016176c104debc759742aa5189983a9ec02b54" Jul 29 16:41:08.879524 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:08.879320 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb" Jul 29 16:41:15.379752 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.379707 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv"] Jul 29 16:41:15.380189 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.379962 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" podUID="cc84d5d2-603b-41df-b29f-56fcea7d5dfe" containerName="maas-api" containerID="cri-o://d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b" gracePeriod=30 Jul 29 16:41:15.583050 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.582988 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb"] Jul 29 16:41:15.585491 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.585460 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3c2006-n4zdb"] Jul 29 16:41:15.641427 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.641354 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:41:15.788621 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.788549 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-openshift-service-ca\") pod \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " Jul 29 16:41:15.788621 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.788621 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-maas-api-tls\") pod \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " Jul 29 16:41:15.788852 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.788685 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcpmn\" (UniqueName: \"kubernetes.io/projected/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-kube-api-access-tcpmn\") pod \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\" (UID: \"cc84d5d2-603b-41df-b29f-56fcea7d5dfe\") " Jul 29 16:41:15.788941 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.788916 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc84d5d2-603b-41df-b29f-56fcea7d5dfe" (UID: "cc84d5d2-603b-41df-b29f-56fcea7d5dfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:15.790841 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.790808 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc84d5d2-603b-41df-b29f-56fcea7d5dfe" (UID: "cc84d5d2-603b-41df-b29f-56fcea7d5dfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:41:15.790952 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.790892 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-kube-api-access-tcpmn" (OuterVolumeSpecName: "kube-api-access-tcpmn") pod "cc84d5d2-603b-41df-b29f-56fcea7d5dfe" (UID: "cc84d5d2-603b-41df-b29f-56fcea7d5dfe"). InnerVolumeSpecName "kube-api-access-tcpmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:15.889972 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.889916 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcpmn\" (UniqueName: \"kubernetes.io/projected/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-kube-api-access-tcpmn\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:15.889972 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.889963 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:15.889972 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.889973 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc84d5d2-603b-41df-b29f-56fcea7d5dfe-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:15.917931 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.917843 2555 generic.go:356] "Generic (PLEG): container finished" podID="cc84d5d2-603b-41df-b29f-56fcea7d5dfe" containerID="d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b" exitCode=0 Jul 29 16:41:15.918073 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.917986 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" Jul 29 16:41:15.925317 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.925281 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d9025c1-d85d-462d-9e62-669541df0845" path="/var/lib/kubelet/pods/0d9025c1-d85d-462d-9e62-669541df0845/volumes" Jul 29 16:41:15.925875 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.925856 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" event={"ID":"cc84d5d2-603b-41df-b29f-56fcea7d5dfe","Type":"ContainerDied","Data":"d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b"} Jul 29 16:41:15.925963 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.925897 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv" event={"ID":"cc84d5d2-603b-41df-b29f-56fcea7d5dfe","Type":"ContainerDied","Data":"63e37fc0192c1bf613cb1b02c920558566810068ea072d84fe75ab1645dc7a9b"} Jul 29 16:41:15.925963 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.925922 2555 scope.go:117] "RemoveContainer" containerID="d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b" Jul 29 16:41:15.935355 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.935337 2555 scope.go:117] "RemoveContainer" containerID="d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b" Jul 29 16:41:15.935668 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:41:15.935637 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b\": container with ID starting with d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b not found: ID does not exist" containerID="d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b" Jul 29 16:41:15.935780 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.935681 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b"} err="failed to get container status \"d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b\": rpc error: code = NotFound desc = could not find container \"d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b\": container with ID starting with d0f7bfcc2bcce513c12f50402d33fbfda661414555c512cec555109bd6c99e9b not found: ID does not exist" Jul 29 16:41:15.945684 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.945653 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv"] Jul 29 16:41:15.951632 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:15.951601 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c2006-5f865947f7-4ksjv"] Jul 29 16:41:17.924720 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:17.924688 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc84d5d2-603b-41df-b29f-56fcea7d5dfe" path="/var/lib/kubelet/pods/cc84d5d2-603b-41df-b29f-56fcea7d5dfe/volumes" Jul 29 16:41:22.639083 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.639042 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6"] Jul 29 16:41:22.639477 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.639426 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d9025c1-d85d-462d-9e62-669541df0845" containerName="revoke-keys" Jul 29 16:41:22.639477 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.639438 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d9025c1-d85d-462d-9e62-669541df0845" containerName="revoke-keys" Jul 29 16:41:22.639477 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.639456 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc84d5d2-603b-41df-b29f-56fcea7d5dfe" containerName="maas-api" Jul 29 16:41:22.639477 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.639461 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc84d5d2-603b-41df-b29f-56fcea7d5dfe" containerName="maas-api" Jul 29 16:41:22.639641 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.639526 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d9025c1-d85d-462d-9e62-669541df0845" containerName="revoke-keys" Jul 29 16:41:22.639641 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.639536 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc84d5d2-603b-41df-b29f-56fcea7d5dfe" containerName="maas-api" Jul 29 16:41:22.643928 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.643909 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" Jul 29 16:41:22.646721 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.646693 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:41:22.649674 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.649646 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6"] Jul 29 16:41:22.754840 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.754801 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f5805c-ea85-4716-95c5-942bdba42de4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6\" (UID: \"a2f5805c-ea85-4716-95c5-942bdba42de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" Jul 29 16:41:22.855555 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.855517 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f5805c-ea85-4716-95c5-942bdba42de4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6\" (UID: \"a2f5805c-ea85-4716-95c5-942bdba42de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" Jul 29 16:41:22.856647 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.856572 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f5805c-ea85-4716-95c5-942bdba42de4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6\" (UID: \"a2f5805c-ea85-4716-95c5-942bdba42de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" Jul 29 16:41:22.956357 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:22.956268 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" Jul 29 16:41:23.099117 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:23.099086 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6"] Jul 29 16:41:23.100765 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:41:23.100741 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f5805c_ea85_4716_95c5_942bdba42de4.slice/crio-de48896b35801427e04fe7c7b749646fa4024a9e0d81028b9c75278f34d74916 WatchSource:0}: Error finding container de48896b35801427e04fe7c7b749646fa4024a9e0d81028b9c75278f34d74916: Status 404 returned error can't find the container with id de48896b35801427e04fe7c7b749646fa4024a9e0d81028b9c75278f34d74916 Jul 29 16:41:23.963155 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:23.963124 2555 generic.go:356] "Generic (PLEG): container finished" podID="a2f5805c-ea85-4716-95c5-942bdba42de4" containerID="c3e4d60473fa337bbe864eecac8eae55323780c58496ca48cbfadaf533140c3a" exitCode=0 Jul 29 16:41:23.963542 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:23.963220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" event={"ID":"a2f5805c-ea85-4716-95c5-942bdba42de4","Type":"ContainerDied","Data":"c3e4d60473fa337bbe864eecac8eae55323780c58496ca48cbfadaf533140c3a"} Jul 29 16:41:23.963542 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:23.963272 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" event={"ID":"a2f5805c-ea85-4716-95c5-942bdba42de4","Type":"ContainerStarted","Data":"de48896b35801427e04fe7c7b749646fa4024a9e0d81028b9c75278f34d74916"} Jul 29 16:41:25.105816 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:25.105783 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" Jul 29 16:41:25.177271 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:25.177235 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f5805c-ea85-4716-95c5-942bdba42de4-maas-api-service-ca\") pod \"a2f5805c-ea85-4716-95c5-942bdba42de4\" (UID: \"a2f5805c-ea85-4716-95c5-942bdba42de4\") " Jul 29 16:41:25.177687 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:25.177665 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f5805c-ea85-4716-95c5-942bdba42de4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2f5805c-ea85-4716-95c5-942bdba42de4" (UID: "a2f5805c-ea85-4716-95c5-942bdba42de4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:25.278852 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:25.278816 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f5805c-ea85-4716-95c5-942bdba42de4-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:25.975054 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:25.975017 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" event={"ID":"a2f5805c-ea85-4716-95c5-942bdba42de4","Type":"ContainerDied","Data":"de48896b35801427e04fe7c7b749646fa4024a9e0d81028b9c75278f34d74916"} Jul 29 16:41:25.975207 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:25.975065 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de48896b35801427e04fe7c7b749646fa4024a9e0d81028b9c75278f34d74916" Jul 29 16:41:25.975207 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:25.975063 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6" Jul 29 16:41:30.162147 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:30.162120 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:41:30.177184 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:30.177158 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:41:33.082723 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.082685 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f"] Jul 29 16:41:33.083211 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.082942 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" podUID="bbe54bf4-8094-4c13-8af6-24348f1e568b" containerName="maas-api" containerID="cri-o://206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87" gracePeriod=30 Jul 29 16:41:33.295395 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.295358 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6"] Jul 29 16:41:33.297391 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.297366 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-81ab8b-9s6d6"] Jul 29 16:41:33.345669 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.345573 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:41:33.455753 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.455704 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe54bf4-8094-4c13-8af6-24348f1e568b-openshift-service-ca\") pod \"bbe54bf4-8094-4c13-8af6-24348f1e568b\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " Jul 29 16:41:33.455753 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.455770 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc99g\" (UniqueName: \"kubernetes.io/projected/bbe54bf4-8094-4c13-8af6-24348f1e568b-kube-api-access-bc99g\") pod \"bbe54bf4-8094-4c13-8af6-24348f1e568b\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " Jul 29 16:41:33.456009 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.455827 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls\") pod \"bbe54bf4-8094-4c13-8af6-24348f1e568b\" (UID: \"bbe54bf4-8094-4c13-8af6-24348f1e568b\") " Jul 29 16:41:33.456203 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.456174 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe54bf4-8094-4c13-8af6-24348f1e568b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbe54bf4-8094-4c13-8af6-24348f1e568b" (UID: "bbe54bf4-8094-4c13-8af6-24348f1e568b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:33.457930 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.457903 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbe54bf4-8094-4c13-8af6-24348f1e568b" (UID: "bbe54bf4-8094-4c13-8af6-24348f1e568b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:41:33.457930 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.457918 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe54bf4-8094-4c13-8af6-24348f1e568b-kube-api-access-bc99g" (OuterVolumeSpecName: "kube-api-access-bc99g") pod "bbe54bf4-8094-4c13-8af6-24348f1e568b" (UID: "bbe54bf4-8094-4c13-8af6-24348f1e568b"). InnerVolumeSpecName "kube-api-access-bc99g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:33.556524 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.556475 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe54bf4-8094-4c13-8af6-24348f1e568b-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:33.556524 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.556521 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc99g\" (UniqueName: \"kubernetes.io/projected/bbe54bf4-8094-4c13-8af6-24348f1e568b-kube-api-access-bc99g\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:33.556524 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.556532 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe54bf4-8094-4c13-8af6-24348f1e568b-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:33.924486 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:33.924451 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f5805c-ea85-4716-95c5-942bdba42de4" path="/var/lib/kubelet/pods/a2f5805c-ea85-4716-95c5-942bdba42de4/volumes" Jul 29 16:41:34.018333 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.018298 2555 generic.go:356] "Generic (PLEG): container finished" podID="bbe54bf4-8094-4c13-8af6-24348f1e568b" containerID="206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87" exitCode=0 Jul 29 16:41:34.018512 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.018347 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" Jul 29 16:41:34.018512 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.018382 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" event={"ID":"bbe54bf4-8094-4c13-8af6-24348f1e568b","Type":"ContainerDied","Data":"206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87"} Jul 29 16:41:34.018512 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.018438 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f" event={"ID":"bbe54bf4-8094-4c13-8af6-24348f1e568b","Type":"ContainerDied","Data":"411b4f5a2b92afd8b5466f4fad1ed0f3eed0736f1010729e88dfcc55ee8c1a96"} Jul 29 16:41:34.018512 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.018460 2555 scope.go:117] "RemoveContainer" containerID="206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87" Jul 29 16:41:34.027950 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.027930 2555 scope.go:117] "RemoveContainer" containerID="206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87" Jul 29 16:41:34.028276 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:41:34.028249 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87\": container with ID starting with 206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87 not found: ID does not exist" containerID="206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87" Jul 29 16:41:34.028389 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.028282 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87"} err="failed to get container status \"206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87\": rpc error: code = NotFound desc = could not find container \"206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87\": container with ID starting with 206d4901dc8c50feb3d4e43d9be5e1a75c3a021cf7216a970356be0728e69c87 not found: ID does not exist" Jul 29 16:41:34.036382 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.036348 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f"] Jul 29 16:41:34.042511 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:34.042481 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-81ab8b-5647f68f44-pvl8f"] Jul 29 16:41:35.924590 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:35.924550 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe54bf4-8094-4c13-8af6-24348f1e568b" path="/var/lib/kubelet/pods/bbe54bf4-8094-4c13-8af6-24348f1e568b/volumes" Jul 29 16:41:40.270863 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.270830 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x"] Jul 29 16:41:40.271353 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.271188 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe54bf4-8094-4c13-8af6-24348f1e568b" containerName="maas-api" Jul 29 16:41:40.271353 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.271199 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe54bf4-8094-4c13-8af6-24348f1e568b" containerName="maas-api" Jul 29 16:41:40.271353 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.271224 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f5805c-ea85-4716-95c5-942bdba42de4" containerName="revoke-keys" Jul 29 16:41:40.271353 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.271231 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f5805c-ea85-4716-95c5-942bdba42de4" containerName="revoke-keys" Jul 29 16:41:40.271353 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.271294 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe54bf4-8094-4c13-8af6-24348f1e568b" containerName="maas-api" Jul 29 16:41:40.271353 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.271305 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f5805c-ea85-4716-95c5-942bdba42de4" containerName="revoke-keys" Jul 29 16:41:40.275842 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.275823 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" Jul 29 16:41:40.278639 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.278614 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:41:40.283287 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.283255 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x"] Jul 29 16:41:40.418907 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.418874 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36fb8e6c-74da-467a-89e4-3bfdcd837708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x\" (UID: \"36fb8e6c-74da-467a-89e4-3bfdcd837708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" Jul 29 16:41:40.520172 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.520131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36fb8e6c-74da-467a-89e4-3bfdcd837708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x\" (UID: \"36fb8e6c-74da-467a-89e4-3bfdcd837708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" Jul 29 16:41:40.520802 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.520782 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36fb8e6c-74da-467a-89e4-3bfdcd837708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x\" (UID: \"36fb8e6c-74da-467a-89e4-3bfdcd837708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" Jul 29 16:41:40.588821 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.588737 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" Jul 29 16:41:40.723381 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:40.723348 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x"] Jul 29 16:41:40.724378 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:41:40.724349 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fb8e6c_74da_467a_89e4_3bfdcd837708.slice/crio-4c33610555aeb75b5f5f41d1baaff75de48684e547dc70aec550fd8e8819d44c WatchSource:0}: Error finding container 4c33610555aeb75b5f5f41d1baaff75de48684e547dc70aec550fd8e8819d44c: Status 404 returned error can't find the container with id 4c33610555aeb75b5f5f41d1baaff75de48684e547dc70aec550fd8e8819d44c Jul 29 16:41:41.058513 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:41.058476 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" event={"ID":"36fb8e6c-74da-467a-89e4-3bfdcd837708","Type":"ContainerStarted","Data":"d410b5c3dec2b4514b0dfa2d0191573bed19f54a86d7e1ee616cd021121f2541"} Jul 29 16:41:41.058735 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:41.058537 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" event={"ID":"36fb8e6c-74da-467a-89e4-3bfdcd837708","Type":"ContainerStarted","Data":"4c33610555aeb75b5f5f41d1baaff75de48684e547dc70aec550fd8e8819d44c"} Jul 29 16:41:41.092559 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:41.092502 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" podStartSLOduration=1.092482765 podStartE2EDuration="1.092482765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:41:41.091141915 +0000 UTC m=+3311.632043384" watchObservedRunningTime="2026-07-29 16:41:41.092482765 +0000 UTC m=+3311.633384233" Jul 29 16:41:42.065774 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:42.065740 2555 generic.go:356] "Generic (PLEG): container finished" podID="36fb8e6c-74da-467a-89e4-3bfdcd837708" containerID="d410b5c3dec2b4514b0dfa2d0191573bed19f54a86d7e1ee616cd021121f2541" exitCode=0 Jul 29 16:41:42.066185 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:42.065826 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" event={"ID":"36fb8e6c-74da-467a-89e4-3bfdcd837708","Type":"ContainerDied","Data":"d410b5c3dec2b4514b0dfa2d0191573bed19f54a86d7e1ee616cd021121f2541"} Jul 29 16:41:43.216811 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:43.216782 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" Jul 29 16:41:43.345850 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:43.345760 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36fb8e6c-74da-467a-89e4-3bfdcd837708-maas-api-service-ca\") pod \"36fb8e6c-74da-467a-89e4-3bfdcd837708\" (UID: \"36fb8e6c-74da-467a-89e4-3bfdcd837708\") " Jul 29 16:41:43.346167 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:43.346146 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fb8e6c-74da-467a-89e4-3bfdcd837708-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36fb8e6c-74da-467a-89e4-3bfdcd837708" (UID: "36fb8e6c-74da-467a-89e4-3bfdcd837708"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:43.446771 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:43.446736 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36fb8e6c-74da-467a-89e4-3bfdcd837708-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:44.078402 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:44.078366 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" event={"ID":"36fb8e6c-74da-467a-89e4-3bfdcd837708","Type":"ContainerDied","Data":"4c33610555aeb75b5f5f41d1baaff75de48684e547dc70aec550fd8e8819d44c"} Jul 29 16:41:44.078402 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:44.078394 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x" Jul 29 16:41:44.078640 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:44.078416 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c33610555aeb75b5f5f41d1baaff75de48684e547dc70aec550fd8e8819d44c" Jul 29 16:41:50.703201 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:50.703165 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv"] Jul 29 16:41:50.703685 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:50.703417 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" podUID="91105a3d-2c55-4f81-9038-12b315b952bf" containerName="maas-api" containerID="cri-o://605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a" gracePeriod=30 Jul 29 16:41:50.907771 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:50.907737 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x"] Jul 29 16:41:50.909772 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:50.909745 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-eccd34-cc97x"] Jul 29 16:41:50.980832 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:50.980805 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:41:51.118817 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.118783 2555 generic.go:356] "Generic (PLEG): container finished" podID="91105a3d-2c55-4f81-9038-12b315b952bf" containerID="605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a" exitCode=0 Jul 29 16:41:51.119027 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.118836 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" event={"ID":"91105a3d-2c55-4f81-9038-12b315b952bf","Type":"ContainerDied","Data":"605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a"} Jul 29 16:41:51.119027 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.118839 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" Jul 29 16:41:51.119027 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.118890 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv" event={"ID":"91105a3d-2c55-4f81-9038-12b315b952bf","Type":"ContainerDied","Data":"d2e54362c1b7f109d74b0ac26fc4edf8874d74dc98c5e62cde7d8b7d854d2dfb"} Jul 29 16:41:51.119027 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.118890 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls\") pod \"91105a3d-2c55-4f81-9038-12b315b952bf\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " Jul 29 16:41:51.119027 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.118965 2555 scope.go:117] "RemoveContainer" containerID="605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a" Jul 29 16:41:51.119027 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.118981 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91105a3d-2c55-4f81-9038-12b315b952bf-openshift-service-ca\") pod \"91105a3d-2c55-4f81-9038-12b315b952bf\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " Jul 29 16:41:51.119342 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.119061 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xxfp\" (UniqueName: \"kubernetes.io/projected/91105a3d-2c55-4f81-9038-12b315b952bf-kube-api-access-7xxfp\") pod \"91105a3d-2c55-4f81-9038-12b315b952bf\" (UID: \"91105a3d-2c55-4f81-9038-12b315b952bf\") " Jul 29 16:41:51.119467 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.119437 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91105a3d-2c55-4f81-9038-12b315b952bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91105a3d-2c55-4f81-9038-12b315b952bf" (UID: "91105a3d-2c55-4f81-9038-12b315b952bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:51.121153 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.121125 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91105a3d-2c55-4f81-9038-12b315b952bf" (UID: "91105a3d-2c55-4f81-9038-12b315b952bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 16:41:51.121316 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.121298 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91105a3d-2c55-4f81-9038-12b315b952bf-kube-api-access-7xxfp" (OuterVolumeSpecName: "kube-api-access-7xxfp") pod "91105a3d-2c55-4f81-9038-12b315b952bf" (UID: "91105a3d-2c55-4f81-9038-12b315b952bf"). InnerVolumeSpecName "kube-api-access-7xxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:51.139833 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.139812 2555 scope.go:117] "RemoveContainer" containerID="605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a" Jul 29 16:41:51.140169 ip-10-0-130-132 kubenswrapper[2555]: E0729 16:41:51.140147 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a\": container with ID starting with 605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a not found: ID does not exist" containerID="605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a" Jul 29 16:41:51.140225 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.140181 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a"} err="failed to get container status \"605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a\": rpc error: code = NotFound desc = could not find container \"605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a\": container with ID starting with 605508a5bd842439dcaa14c444942f1b72d34165a381ac3b9cfb97683fa3601a not found: ID does not exist" Jul 29 16:41:51.220442 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.220353 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91105a3d-2c55-4f81-9038-12b315b952bf-openshift-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:51.220442 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.220384 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xxfp\" (UniqueName: \"kubernetes.io/projected/91105a3d-2c55-4f81-9038-12b315b952bf-kube-api-access-7xxfp\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:51.220442 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.220394 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91105a3d-2c55-4f81-9038-12b315b952bf-maas-api-tls\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\"" Jul 29 16:41:51.440935 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.440894 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv"] Jul 29 16:41:51.444678 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.444652 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-eccd34-674d4444f7-7s7kv"] Jul 29 16:41:51.924741 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.924709 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36fb8e6c-74da-467a-89e4-3bfdcd837708" path="/var/lib/kubelet/pods/36fb8e6c-74da-467a-89e4-3bfdcd837708/volumes" Jul 29 16:41:51.925116 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:41:51.925103 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91105a3d-2c55-4f81-9038-12b315b952bf" path="/var/lib/kubelet/pods/91105a3d-2c55-4f81-9038-12b315b952bf/volumes" Jul 29 16:42:04.325070 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:04.325039 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-mkm2b_8d95a3de-1277-4bb4-9668-606f736ca6c1/copy-manifests/0.log" Jul 29 16:42:04.331065 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:04.331042 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-mkm2b_8d95a3de-1277-4bb4-9668-606f736ca6c1/manager/0.log" Jul 29 16:42:04.841890 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:04.841859 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79554c65f8-67w2q_efa44fd7-5cf9-4b92-80cd-cc8ddecb0cf7/maas-api/0.log" Jul 29 16:42:05.292917 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:05.292885 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5k6qg_008fc234-a584-490d-8faa-0abda8684e1d/postgres/0.log" Jul 29 16:42:06.049621 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.049567 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2_e9c3688c-416b-4570-b346-7c7f8713a69a/util/0.log" Jul 29 16:42:06.055574 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.055545 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2_e9c3688c-416b-4570-b346-7c7f8713a69a/pull/0.log" Jul 29 16:42:06.061335 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.061314 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2_e9c3688c-416b-4570-b346-7c7f8713a69a/extract/0.log" Jul 29 16:42:06.289758 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.289729 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd_916f64ed-b951-416d-8f90-4d2243c2c7be/util/0.log" Jul 29 16:42:06.295471 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.295451 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd_916f64ed-b951-416d-8f90-4d2243c2c7be/pull/0.log" Jul 29 16:42:06.300535 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.300485 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd_916f64ed-b951-416d-8f90-4d2243c2c7be/extract/0.log" Jul 29 16:42:06.408904 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.408878 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5_cdb23bd3-186e-4ba1-b8ae-359f9afd4d21/util/0.log" Jul 29 16:42:06.415011 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.414989 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5_cdb23bd3-186e-4ba1-b8ae-359f9afd4d21/pull/0.log" Jul 29 16:42:06.421362 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:06.421341 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5_cdb23bd3-186e-4ba1-b8ae-359f9afd4d21/extract/0.log" Jul 29 16:42:07.830481 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:07.830449 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hlz67_8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a/discovery/0.log" Jul 29 16:42:08.292323 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:08.292291 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mvwtl_1cbc2197-3336-4a4c-83d1-d7dd99c6d13f/payload-pre-processing/0.log" Jul 29 16:42:08.411258 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:08.411229 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hpvlw_cc9cbd65-0255-4ed1-83ad-3d534ec9d42d/payload-processing/0.log" Jul 29 16:42:08.527254 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:08.527227 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-658f857f8f-j4knh_edc2889a-6586-468b-9fa8-f7d8ff946770/router/0.log" Jul 29 16:42:09.087410 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:09.087382 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-s7287_28799df6-819e-480a-bccc-fdae883212a0/main/0.log" Jul 29 16:42:09.306556 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:09.306528 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tkpdc_0c83ed8a-e20c-4aac-a807-16724d4a9150/main/0.log" Jul 29 16:42:17.853885 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:17.853847 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ptml9_c6d0e693-ccf2-4168-a63c-2f1428af95fe/global-pull-secret-syncer/0.log" Jul 29 16:42:18.015723 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:18.015689 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-skhtl_69cf0189-a6a2-406d-991b-35b69c360c09/konnectivity-agent/0.log" Jul 29 16:42:18.034724 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:18.034695 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-132.ec2.internal_a573c2ec0c2e9c2aaedfbe717cd64a75/haproxy/0.log" Jul 29 16:42:22.558512 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.558481 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2_e9c3688c-416b-4570-b346-7c7f8713a69a/extract/0.log" Jul 29 16:42:22.582050 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.582013 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2_e9c3688c-416b-4570-b346-7c7f8713a69a/util/0.log" Jul 29 16:42:22.604967 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.604937 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wf2g2_e9c3688c-416b-4570-b346-7c7f8713a69a/pull/0.log" Jul 29 16:42:22.743699 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.743670 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd_916f64ed-b951-416d-8f90-4d2243c2c7be/extract/0.log" Jul 29 16:42:22.765475 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.765444 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd_916f64ed-b951-416d-8f90-4d2243c2c7be/util/0.log" Jul 29 16:42:22.792129 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.792103 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734kggd_916f64ed-b951-416d-8f90-4d2243c2c7be/pull/0.log" Jul 29 16:42:22.820830 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.820762 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5_cdb23bd3-186e-4ba1-b8ae-359f9afd4d21/extract/0.log" Jul 29 16:42:22.841199 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.841175 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5_cdb23bd3-186e-4ba1-b8ae-359f9afd4d21/util/0.log" Jul 29 16:42:22.861373 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:22.861345 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qkcg5_cdb23bd3-186e-4ba1-b8ae-359f9afd4d21/pull/0.log" Jul 29 16:42:25.953526 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:25.953499 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-8hm8n_cabeca35-f4d0-4626-82d2-be6ded895a50/cluster-monitoring-operator/0.log" Jul 29 16:42:25.979066 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:25.979038 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zbwvb_09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b/kube-state-metrics/0.log" Jul 29 16:42:25.997431 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:25.997393 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zbwvb_09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b/kube-rbac-proxy-main/0.log" Jul 29 16:42:26.017077 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:26.017049 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zbwvb_09fa9345-a3af-4cf6-a0ac-e829f6ac6d2b/kube-rbac-proxy-self/0.log" Jul 29 16:42:26.252234 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:26.252153 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tshzs_b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9/node-exporter/0.log" Jul 29 16:42:26.271682 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:26.271643 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tshzs_b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9/kube-rbac-proxy/0.log" Jul 29 16:42:26.292306 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:26.292276 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tshzs_b364fbdd-b38b-46e7-ab05-7d3e8a1b84c9/init-textfile/0.log" Jul 29 16:42:26.601897 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:26.601867 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d5bf5d8f4-tf5fd_154d0538-6942-4b62-a2a0-26b36fb0d7b6/telemeter-client/0.log" Jul 29 16:42:26.621819 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:26.621794 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d5bf5d8f4-tf5fd_154d0538-6942-4b62-a2a0-26b36fb0d7b6/reload/0.log" Jul 29 16:42:26.649884 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:26.649859 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d5bf5d8f4-tf5fd_154d0538-6942-4b62-a2a0-26b36fb0d7b6/kube-rbac-proxy/0.log" Jul 29 16:42:27.902925 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.902890 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9"] Jul 29 16:42:27.903333 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.903236 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91105a3d-2c55-4f81-9038-12b315b952bf" containerName="maas-api" Jul 29 16:42:27.903333 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.903246 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91105a3d-2c55-4f81-9038-12b315b952bf" containerName="maas-api" Jul 29 16:42:27.903333 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.903269 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fb8e6c-74da-467a-89e4-3bfdcd837708" containerName="revoke-keys" Jul 29 16:42:27.903333 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.903275 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fb8e6c-74da-467a-89e4-3bfdcd837708" containerName="revoke-keys" Jul 29 16:42:27.903481 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.903337 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="36fb8e6c-74da-467a-89e4-3bfdcd837708" containerName="revoke-keys" Jul 29 16:42:27.903481 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.903347 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91105a3d-2c55-4f81-9038-12b315b952bf" containerName="maas-api" Jul 29 16:42:27.906723 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.906706 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:27.909750 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.909728 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqvb7\"/\"openshift-service-ca.crt\"" Jul 29 16:42:27.910035 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.910021 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqvb7\"/\"kube-root-ca.crt\"" Jul 29 16:42:27.910939 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.910920 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kqvb7\"/\"default-dockercfg-hb6xm\"" Jul 29 16:42:27.915165 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.915141 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9"] Jul 29 16:42:27.949285 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.949240 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-lib-modules\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:27.949285 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.949290 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-proc\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:27.949532 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.949325 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-sys\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:27.949532 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.949348 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk89n\" (UniqueName: \"kubernetes.io/projected/baae84b4-5ee8-428d-a19c-c7bf0b53014e-kube-api-access-wk89n\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:27.949532 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.949382 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-podres\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:27.998904 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:27.998877 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-t5f2r_cb906de3-e1b3-4082-a332-b199c3fecccb/networking-console-plugin/0.log" Jul 29 16:42:28.050348 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050307 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-proc\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050348 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050355 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-sys\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050628 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050374 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk89n\" (UniqueName: \"kubernetes.io/projected/baae84b4-5ee8-428d-a19c-c7bf0b53014e-kube-api-access-wk89n\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050628 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050415 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-podres\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050628 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050437 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-proc\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050628 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-sys\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050628 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050537 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-lib-modules\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050628 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050571 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-podres\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.050908 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.050711 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baae84b4-5ee8-428d-a19c-c7bf0b53014e-lib-modules\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.059784 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.059746 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk89n\" (UniqueName: \"kubernetes.io/projected/baae84b4-5ee8-428d-a19c-c7bf0b53014e-kube-api-access-wk89n\") pod \"perf-node-gather-daemonset-gn6h9\" (UID: \"baae84b4-5ee8-428d-a19c-c7bf0b53014e\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.217239 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.217144 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:28.357528 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.357494 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9"] Jul 29 16:42:28.361255 ip-10-0-130-132 kubenswrapper[2555]: W0729 16:42:28.361218 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbaae84b4_5ee8_428d_a19c_c7bf0b53014e.slice/crio-15f740ff3916f06087a7517018f84bf080f6344baa6a54a7a9edb7cdb3bcd8fd WatchSource:0}: Error finding container 15f740ff3916f06087a7517018f84bf080f6344baa6a54a7a9edb7cdb3bcd8fd: Status 404 returned error can't find the container with id 15f740ff3916f06087a7517018f84bf080f6344baa6a54a7a9edb7cdb3bcd8fd Jul 29 16:42:28.625055 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.625025 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/1.log" Jul 29 16:42:28.629937 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:28.629909 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-6mrvv_d0e2ec2f-3b47-4eb1-99b4-eb44bc7e5bec/console-operator/2.log" Jul 29 16:42:29.051083 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:29.051056 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76bb745dd7-tm5qq_fc6ef919-cb3d-4918-9c30-a7c2319087ef/console/0.log" Jul 29 16:42:29.326900 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:29.326807 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" event={"ID":"baae84b4-5ee8-428d-a19c-c7bf0b53014e","Type":"ContainerStarted","Data":"9c375a951a3db0bcbc5a17eec19d4787cfb418f99f856cfff5595d6544dbcb99"} Jul 29 16:42:29.326900 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:29.326886 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" event={"ID":"baae84b4-5ee8-428d-a19c-c7bf0b53014e","Type":"ContainerStarted","Data":"15f740ff3916f06087a7517018f84bf080f6344baa6a54a7a9edb7cdb3bcd8fd"} Jul 29 16:42:29.327127 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:29.326939 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:29.344741 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:29.344684 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" podStartSLOduration=2.344667725 podStartE2EDuration="2.344667725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:42:29.341669041 +0000 UTC m=+3359.882570511" watchObservedRunningTime="2026-07-29 16:42:29.344667725 +0000 UTC m=+3359.885569254" Jul 29 16:42:29.631655 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:29.631549 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-2fhqc_8d1827c0-2505-404a-8af6-e386069efca2/volume-data-source-validator/0.log" Jul 29 16:42:30.696443 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:30.696411 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmzml_571cee9d-30cc-4291-91c0-abb9ef90f3c5/dns/0.log" Jul 29 16:42:30.715142 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:30.715114 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmzml_571cee9d-30cc-4291-91c0-abb9ef90f3c5/kube-rbac-proxy/0.log" Jul 29 16:42:30.778254 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:30.778227 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m7jv2_be264cf7-2ed3-4c99-8a85-264cacb645c7/dns-node-resolver/0.log" Jul 29 16:42:31.360128 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:31.360096 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6mvd9_120f7d41-d050-48d7-acd8-c9779c7af52f/node-ca/0.log" Jul 29 16:42:32.444110 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:32.444076 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hlz67_8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a/discovery/0.log" Jul 29 16:42:32.597877 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:32.597846 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mvwtl_1cbc2197-3336-4a4c-83d1-d7dd99c6d13f/payload-pre-processing/0.log" Jul 29 16:42:32.622919 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:32.622884 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hpvlw_cc9cbd65-0255-4ed1-83ad-3d534ec9d42d/payload-processing/0.log" Jul 29 16:42:32.641785 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:32.641756 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-658f857f8f-j4knh_edc2889a-6586-468b-9fa8-f7d8ff946770/router/0.log" Jul 29 16:42:33.439513 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:33.439485 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l7g6v_846b9e1c-7ded-4a38-86ad-61f753398cf6/serve-healthcheck-canary/0.log" Jul 29 16:42:33.919056 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:33.919021 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-vrsmf_cba9d9bd-cb1c-48e0-a19d-46c76e955a4f/insights-operator/0.log" Jul 29 16:42:33.919860 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:33.919841 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-vrsmf_cba9d9bd-cb1c-48e0-a19d-46c76e955a4f/insights-operator/1.log" Jul 29 16:42:34.004391 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:34.004363 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p4n8x_794e5299-4224-4873-9c41-6857ad5c2390/kube-rbac-proxy/0.log" Jul 29 16:42:34.025251 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:34.025225 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p4n8x_794e5299-4224-4873-9c41-6857ad5c2390/exporter/0.log" Jul 29 16:42:34.046933 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:34.046906 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p4n8x_794e5299-4224-4873-9c41-6857ad5c2390/extractor/0.log" Jul 29 16:42:35.342511 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:35.342485 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-gn6h9" Jul 29 16:42:37.299710 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:37.299668 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-mkm2b_8d95a3de-1277-4bb4-9668-606f736ca6c1/manager/0.log" Jul 29 16:42:37.320896 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:37.320874 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-mkm2b_8d95a3de-1277-4bb4-9668-606f736ca6c1/copy-manifests/0.log" Jul 29 16:42:38.139866 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:38.139778 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hlz67_8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a/discovery/0.log" Jul 29 16:42:38.259953 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:38.259923 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mvwtl_1cbc2197-3336-4a4c-83d1-d7dd99c6d13f/payload-pre-processing/0.log" Jul 29 16:42:38.277900 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:38.277869 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hpvlw_cc9cbd65-0255-4ed1-83ad-3d534ec9d42d/payload-processing/0.log" Jul 29 16:42:38.290443 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:38.290417 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-658f857f8f-j4knh_edc2889a-6586-468b-9fa8-f7d8ff946770/router/0.log" Jul 29 16:42:38.711285 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:38.711261 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cb94b984b-q7qjp_959ab688-2096-4d1a-a44f-aa18d790afe4/manager/0.log" Jul 29 16:42:39.827996 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:39.827961 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hlz67_8ca8c4b0-cc3e-4516-b4a5-1111fdfbb11a/discovery/0.log" Jul 29 16:42:39.948932 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:39.948903 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-mvwtl_1cbc2197-3336-4a4c-83d1-d7dd99c6d13f/payload-pre-processing/0.log" Jul 29 16:42:39.969539 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:39.969508 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hpvlw_cc9cbd65-0255-4ed1-83ad-3d534ec9d42d/payload-processing/0.log" Jul 29 16:42:39.984100 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:39.984072 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-658f857f8f-j4knh_edc2889a-6586-468b-9fa8-f7d8ff946770/router/0.log" Jul 29 16:42:43.744616 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:43.744561 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp_b18913da-9faf-4eb2-bfbf-f63e9308022e/kube-storage-version-migrator-operator/1.log" Jul 29 16:42:43.745381 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:43.745363 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nk9kp_b18913da-9faf-4eb2-bfbf-f63e9308022e/kube-storage-version-migrator-operator/0.log" Jul 29 16:42:44.713645 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:44.713613 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8pzdz_8341f0c8-f991-4c19-a797-f7fce085608f/kube-multus-additional-cni-plugins/0.log" Jul 29 16:42:44.732523 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:44.732491 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8pzdz_8341f0c8-f991-4c19-a797-f7fce085608f/egress-router-binary-copy/0.log" Jul 29 16:42:44.750993 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:44.750968 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8pzdz_8341f0c8-f991-4c19-a797-f7fce085608f/cni-plugins/0.log" Jul 29 16:42:44.769365 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:44.769335 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8pzdz_8341f0c8-f991-4c19-a797-f7fce085608f/bond-cni-plugin/0.log" Jul 29 16:42:44.787691 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:44.787660 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8pzdz_8341f0c8-f991-4c19-a797-f7fce085608f/routeoverride-cni/0.log" Jul 29 16:42:44.807110 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:44.807078 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8pzdz_8341f0c8-f991-4c19-a797-f7fce085608f/whereabouts-cni-bincopy/0.log" Jul 29 16:42:44.825912 ip-10-0-130-132 kubenswrapper[2555]: I0729 16:42:44.825880 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8pzdz_8341f0c8-f991-4c19-a797-f7fce085608f/whereabouts-cni/0.log"