Jul 10 22:42:59.703483 ip-10-0-139-196 systemd[1]: Starting Kubernetes Kubelet... Jul 10 22:43:00.058086 ip-10-0-139-196 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:43:00.058086 ip-10-0-139-196 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 22:43:00.058086 ip-10-0-139-196 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:43:00.058086 ip-10-0-139-196 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 22:43:00.058086 ip-10-0-139-196 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:43:00.059658 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.059542 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 22:43:00.065443 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065420 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:43:00.065443 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065438 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:43:00.065443 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065442 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:43:00.065443 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065447 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065451 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065454 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065457 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065460 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065462 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065465 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065468 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065471 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065474 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065476 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065479 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065482 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065484 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065487 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065491 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065494 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065497 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065500 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:43:00.065611 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065502 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065505 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065507 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065512 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065515 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065519 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065521 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065524 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065528 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065531 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065534 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065536 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065539 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065542 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065546 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065549 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065552 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065555 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065557 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:43:00.066196 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065560 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065563 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065565 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065568 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065571 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065574 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065577 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065580 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065582 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065585 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065588 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065590 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065593 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065596 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065598 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065601 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065604 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065606 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065609 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065612 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:43:00.066658 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065614 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065617 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065620 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065622 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065625 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065628 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065631 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065634 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065638 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065640 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065643 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065647 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065650 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065653 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065655 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065658 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065661 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065663 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065666 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065668 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:43:00.067143 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065671 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065673 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065676 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065679 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065681 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065684 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065687 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065690 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065692 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065695 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065697 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065700 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065703 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065705 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065710 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065713 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065716 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.065718 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066473 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066482 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:43:00.067634 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066486 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066490 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066493 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066497 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066500 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066503 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066505 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066508 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066511 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066514 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066516 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066519 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066521 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066524 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066527 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066530 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066533 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066535 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066538 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066541 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:43:00.068167 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066544 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066546 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066549 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066552 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066554 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066557 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066559 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066562 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066564 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066567 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066569 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066572 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066575 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066578 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066580 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066583 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066585 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066588 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066590 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066593 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:43:00.068690 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066595 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066598 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066600 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066603 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066606 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066608 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066611 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066614 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066617 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066620 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066623 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066626 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066629 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066631 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066634 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066637 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066639 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066642 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066644 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066649 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:43:00.069184 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066653 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066655 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066658 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066662 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066665 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066668 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066670 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066673 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066675 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066678 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066681 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066684 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066686 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066688 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066691 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066694 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066696 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066699 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066701 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066705 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:43:00.069666 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066707 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066710 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066713 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066719 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066723 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066725 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066729 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066733 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066736 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066740 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066743 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066746 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066748 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066751 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066753 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066756 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.066758 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067124 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:43:00.070152 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067144 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067148 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067152 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067155 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067158 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067161 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067164 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067168 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067172 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067175 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067178 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067181 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067184 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067187 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067190 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067194 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067197 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067200 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067203 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067205 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:43:00.070583 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067208 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067210 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067214 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067216 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067219 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067221 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067224 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067227 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067229 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067232 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067234 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067237 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067240 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067242 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067245 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067247 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067251 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067254 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067257 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:43:00.071055 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067260 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067263 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067265 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067268 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067271 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067274 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067279 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067281 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067284 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067288 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067290 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067293 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067295 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067298 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067300 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067303 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067305 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067308 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067311 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067313 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:43:00.071535 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067316 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067318 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067321 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067323 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067326 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067328 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067331 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067334 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067336 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067339 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067341 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067344 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067346 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067350 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067353 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067355 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067357 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067360 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067364 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067367 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:43:00.072019 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067370 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067372 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067375 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067378 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067380 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067383 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067385 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067388 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067390 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067393 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067395 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067398 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067400 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067403 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067405 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067408 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067411 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067413 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.067416 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067478 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067485 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 22:43:00.072512 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067492 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067497 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067500 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067504 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067508 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067512 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067515 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067518 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067522 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067527 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067530 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067533 2577 flags.go:64] FLAG: --cgroup-root="" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067536 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067539 2577 flags.go:64] FLAG: --client-ca-file="" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067543 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067546 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067551 2577 flags.go:64] FLAG: --cluster-domain="" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067553 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067557 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067560 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067564 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067567 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067570 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 22:43:00.073030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067573 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067576 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067580 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067583 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067586 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067589 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067592 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067597 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067600 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067603 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067606 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067609 2577 flags.go:64] FLAG: --enable-server="true" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067612 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067616 2577 flags.go:64] FLAG: --event-burst="100" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067619 2577 flags.go:64] FLAG: --event-qps="50" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067622 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067625 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067628 2577 flags.go:64] FLAG: --eviction-hard="" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067633 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067636 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067640 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067643 2577 flags.go:64] FLAG: --eviction-soft="" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067646 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067649 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067653 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 22:43:00.073602 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067656 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067659 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067662 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067665 2577 flags.go:64] FLAG: --feature-gates="" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067669 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067672 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067675 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067678 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067681 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067684 2577 flags.go:64] FLAG: --help="false" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067687 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-196.ec2.internal" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067691 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067694 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067697 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067700 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067703 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067706 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067710 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067712 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067715 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067718 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067721 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067724 2577 flags.go:64] FLAG: --kube-reserved="" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067727 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 22:43:00.074257 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067730 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067735 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067738 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067741 2577 flags.go:64] FLAG: --lock-file="" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067744 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067747 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067750 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067755 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067758 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067761 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067764 2577 flags.go:64] FLAG: --logging-format="text" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067767 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067770 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067773 2577 flags.go:64] FLAG: --manifest-url="" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067776 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067780 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067783 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067787 2577 flags.go:64] FLAG: --max-pods="110" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067790 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067793 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067796 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067799 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067802 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067805 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067808 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 22:43:00.074833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067815 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067819 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067822 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067825 2577 flags.go:64] FLAG: --pod-cidr="" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067828 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067834 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067838 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067841 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067845 2577 flags.go:64] FLAG: --port="10250" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067848 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067851 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b5b22560dd772565" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067855 2577 flags.go:64] FLAG: --qos-reserved="" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067858 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067861 2577 flags.go:64] FLAG: --register-node="true" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067863 2577 flags.go:64] FLAG: --register-with-taints="" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067873 2577 flags.go:64] FLAG: --registry-burst="10" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067876 2577 flags.go:64] FLAG: --registry-qps="5" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067879 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067882 2577 flags.go:64] FLAG: --reserved-memory="" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067886 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067889 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067892 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067895 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067898 2577 flags.go:64] FLAG: --runonce="false" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067901 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 22:43:00.075702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067904 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067907 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067910 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067913 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067916 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067919 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067922 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067925 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067928 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067931 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067934 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067937 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067940 2577 flags.go:64] FLAG: --system-cgroups="" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067944 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067949 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067953 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067956 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067960 2577 flags.go:64] FLAG: --tls-min-version="" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067963 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067966 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067969 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067971 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067974 2577 flags.go:64] FLAG: --v="2" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067978 2577 flags.go:64] FLAG: --version="false" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067983 2577 flags.go:64] FLAG: --vmodule="" Jul 10 22:43:00.076767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067986 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.067990 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068108 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068112 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068115 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068118 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068121 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068124 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068140 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068143 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068145 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068148 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068151 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068154 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068157 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068160 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068162 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068165 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068168 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068170 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:43:00.077851 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068175 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068177 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068182 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068185 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068188 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068191 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068193 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068196 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068199 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068201 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068204 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068207 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068210 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068213 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068216 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068218 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068221 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068225 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068228 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068231 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:43:00.078355 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068235 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068238 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068241 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068243 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068246 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068249 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068252 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068254 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068257 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068260 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068262 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068265 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068269 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068271 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068276 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068278 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068281 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068284 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068286 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:43:00.079217 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068289 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068291 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068294 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068297 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068300 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068303 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068306 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068308 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068311 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068313 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068316 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068319 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068321 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068324 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068326 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068330 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068334 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068337 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068340 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:43:00.080002 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068342 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068345 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068348 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068350 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068353 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068356 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068359 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068362 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068367 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068370 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068372 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068374 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068377 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068380 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068383 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068385 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068388 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068391 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068394 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068396 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:43:00.080576 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068399 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068402 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.068405 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.069144 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.077737 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.077757 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.078445 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.078459 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078549 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078557 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078562 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078567 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078572 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078576 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078581 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:43:00.081411 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078586 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078591 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078595 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078599 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078604 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078608 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078612 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078617 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078621 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078625 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078629 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078633 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078637 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078642 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078646 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078651 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078655 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078659 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078663 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078669 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:43:00.081871 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078673 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078677 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078683 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078687 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078691 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078698 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078703 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078708 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078712 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078719 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078725 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078729 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078734 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078738 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078742 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078746 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078751 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078755 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078759 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:43:00.082430 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078763 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078767 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078772 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078776 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078780 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078785 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078789 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078794 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078798 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078802 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078807 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078812 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078816 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078820 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078824 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078829 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078834 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078838 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078843 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078847 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:43:00.082972 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078851 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078856 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078861 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078865 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078869 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078873 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078878 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078882 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078886 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078890 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078894 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078898 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078902 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078906 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078913 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078918 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078923 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078928 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078933 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078938 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:43:00.083596 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078942 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078947 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078951 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078955 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078960 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078964 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078968 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078973 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078977 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078982 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078987 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078991 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.078995 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.079003 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079209 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:43:00.084101 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079219 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079224 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079229 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079233 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079238 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079243 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079247 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079252 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079257 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079261 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079266 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079272 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079277 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079281 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079287 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079292 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079296 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079301 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079308 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:43:00.084496 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079314 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079319 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079323 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079327 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079332 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079336 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079342 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079346 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079351 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079356 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079360 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079364 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079368 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079372 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079376 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079380 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079385 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079389 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079393 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079398 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:43:00.085030 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079402 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079406 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079410 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079415 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079419 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079424 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079428 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079432 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079436 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079440 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079447 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079452 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079456 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079460 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079465 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079469 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079473 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079477 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079482 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079486 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:43:00.085578 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079490 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079495 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079499 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079503 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079508 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079512 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079517 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079521 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079525 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079529 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079533 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079537 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079543 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079548 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079553 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079558 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079563 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079568 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079572 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:43:00.086071 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079577 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079582 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079586 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079591 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079595 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079600 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079604 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079608 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079612 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079616 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079621 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079625 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079629 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079634 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079638 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079645 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079649 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079653 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079657 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:43:00.086760 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:00.079661 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:43:00.087538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.079669 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:43:00.087538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.079828 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 22:43:00.087538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.083216 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 22:43:00.087538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.084666 2577 server.go:1019] "Starting client certificate rotation" Jul 10 22:43:00.087538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.084772 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 22:43:00.087538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.084842 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 22:43:00.087538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.087233 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 22:43:00.091935 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.091915 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 22:43:00.102687 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.102667 2577 log.go:25] "Validated CRI v1 runtime API" Jul 10 22:43:00.104662 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.104647 2577 log.go:25] "Validated CRI v1 image API" Jul 10 22:43:00.105530 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.105513 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 22:43:00.109916 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.109898 2577 fs.go:135] Filesystem UUIDs: map[4f205642-59d6-4ea0-9d30-380792e61f68:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 836563c8-2852-495d-86e8-dad31c6fc1f2:/dev/nvme0n1p3] Jul 10 22:43:00.109986 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.109916 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 22:43:00.112757 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.112737 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 22:43:00.115015 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.114903 2577 manager.go:216] Machine: {Timestamp:2026-07-10 22:43:00.114400528 +0000 UTC m=+0.304933450 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107032 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29a1794550caef98af338d895efa5b SystemUUID:ec29a179-4550-caef-98af-338d895efa5b BootID:4256b886-f978-444d-8864-8da5e8e50b2a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:43:d0:5e:b7:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:43:d0:5e:b7:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:0e:c9:31:6c:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 22:43:00.115015 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.115007 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 22:43:00.115152 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.115082 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 22:43:00.118588 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.118559 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 22:43:00.118742 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.118590 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 22:43:00.119839 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.119828 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 22:43:00.119870 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.119841 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 22:43:00.119870 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.119848 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 22:43:00.120492 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.120481 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 22:43:00.121766 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.121757 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 22:43:00.125604 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.125593 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 10 22:43:00.125715 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.125705 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 22:43:00.125746 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.125734 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 10 22:43:00.125746 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.125744 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 22:43:00.125802 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.125756 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 22:43:00.125802 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.125764 2577 kubelet.go:398] "Adding apiserver pod source" Jul 10 22:43:00.125802 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.125778 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 22:43:00.126150 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.126122 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 22:43:00.126190 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.126166 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 22:43:00.128154 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.128124 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 22:43:00.130754 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.130735 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 22:43:00.130834 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.130778 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 22:43:00.130979 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.130967 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 22:43:00.131030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.130987 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 22:43:00.131030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.130996 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 22:43:00.131030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131004 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 22:43:00.131030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131015 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 22:43:00.131030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131023 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 22:43:00.131563 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131543 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 22:43:00.131563 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131561 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 22:43:00.131718 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131581 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 22:43:00.131718 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131591 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 22:43:00.131718 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131605 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 22:43:00.131718 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.131619 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 22:43:00.132472 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.132450 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 22:43:00.132535 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.132487 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 22:43:00.136253 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.136239 2577 server.go:1268] "Started kubelet" Jul 10 22:43:00.136505 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.136454 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 22:43:00.136585 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.136520 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 22:43:00.136854 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.136838 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 22:43:00.137022 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:00.136997 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 22:43:00.137115 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:00.136997 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 22:43:00.137115 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.137004 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 22:43:00.137173 ip-10-0-139-196 systemd[1]: Started Kubernetes Kubelet. Jul 10 22:43:00.138059 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.137944 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 22:43:00.139699 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.139685 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 10 22:43:00.141248 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:00.141230 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 22:43:00.141854 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:00.140587 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-196.ec2.internal.18c10ef6b293e811 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-196.ec2.internal,UID:ip-10-0-139-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-196.ec2.internal,},FirstTimestamp:2026-07-10 22:43:00.136208401 +0000 UTC m=+0.326741300,LastTimestamp:2026-07-10 22:43:00.136208401 +0000 UTC m=+0.326741300,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-196.ec2.internal,}" Jul 10 22:43:00.144222 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.144208 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 22:43:00.144265 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.144214 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 22:43:00.144314 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:00.144281 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:00.144958 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.144932 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 22:43:00.145052 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.144965 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 22:43:00.145052 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145007 2577 factory.go:55] Registering systemd factory Jul 10 22:43:00.145052 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145020 2577 factory.go:223] Registration of the systemd container factory successfully Jul 10 22:43:00.145230 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.144931 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 22:43:00.145230 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145084 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 10 22:43:00.145230 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145103 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 10 22:43:00.145333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145282 2577 factory.go:153] Registering CRI-O factory Jul 10 22:43:00.145333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145297 2577 factory.go:223] Registration of the crio container factory successfully Jul 10 22:43:00.145412 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145368 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 22:43:00.145412 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145394 2577 factory.go:103] Registering Raw factory Jul 10 22:43:00.145412 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.145409 2577 manager.go:1196] Started watching for new ooms in manager Jul 10 22:43:00.146297 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.146283 2577 manager.go:349] Starting recovery of all containers Jul 10 22:43:00.146656 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:00.146629 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 22:43:00.146740 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:00.146720 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 22:43:00.156923 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.156900 2577 manager.go:354] Recovery completed Jul 10 22:43:00.157757 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.157734 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kq6fg" Jul 10 22:43:00.161899 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.161886 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:43:00.163622 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.163608 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:43:00.163692 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.163634 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:43:00.163692 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.163644 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:43:00.164237 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.164223 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:43:00.164292 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.164223 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 22:43:00.164292 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.164254 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 22:43:00.164292 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.164273 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 10 22:43:00.164859 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.164844 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:43:00.164936 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.164870 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:43:00.164936 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.164884 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:43:00.166403 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.166392 2577 policy_none.go:49] "None policy: Start" Jul 10 22:43:00.166445 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.166408 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 22:43:00.166445 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.166419 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 22:43:00.167241 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.167227 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kq6fg" Jul 10 22:43:00.168347 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:00.168336 2577 policy_none.go:47] "Start" Jul 10 22:43:01.514665 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.514515 2577 manager.go:337] "Starting Device Plugin manager" Jul 10 22:43:01.514777 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.514680 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 22:43:01.514777 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.514695 2577 server.go:87] "Starting device plugin registration server" Jul 10 22:43:01.514922 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.514910 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 22:43:01.515019 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.514923 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 22:43:01.515071 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.515028 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 22:43:01.515150 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.515134 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 22:43:01.515150 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.515146 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 22:43:01.515643 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.515626 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 22:43:01.515721 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.515669 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:01.615431 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.615386 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:43:01.617562 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.616428 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:43:01.617562 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.616456 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:43:01.617562 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.616467 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:43:01.617562 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.616502 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.625636 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.625593 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.625636 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.625619 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-196.ec2.internal\": node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:01.627692 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.627665 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 22:43:01.629004 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.628986 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 22:43:01.629109 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.629029 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 22:43:01.629109 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.629055 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 22:43:01.629109 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.629094 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 22:43:01.631180 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.631161 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:43:01.641717 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.641695 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:01.729470 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.729433 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal"] Jul 10 22:43:01.729615 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.729508 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:43:01.731292 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.731276 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:43:01.731380 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.731304 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:43:01.731380 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.731315 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:43:01.733542 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.733530 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:43:01.733672 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.733656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.733709 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.733687 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:43:01.734453 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.734438 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:43:01.734516 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.734465 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:43:01.734516 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.734475 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:43:01.734845 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.734828 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:43:01.734886 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.734858 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:43:01.734886 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.734867 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:43:01.736619 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.736606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.736669 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.736630 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:43:01.739397 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.738746 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:43:01.739397 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.738772 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:43:01.739397 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.738784 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:43:01.742612 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.742596 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:01.762591 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.762567 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-196.ec2.internal\" not found" node="ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.766739 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.766723 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-196.ec2.internal\" not found" node="ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.843333 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.843310 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:01.859905 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.859877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39cf6f2e19ecbd85104d624d7c507309-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal\" (UID: \"39cf6f2e19ecbd85104d624d7c507309\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.860012 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.859912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39cf6f2e19ecbd85104d624d7c507309-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal\" (UID: \"39cf6f2e19ecbd85104d624d7c507309\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.860012 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.859937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/196243ffdd7700b8404231b37ec9c062-config\") pod \"kube-apiserver-proxy-ip-10-0-139-196.ec2.internal\" (UID: \"196243ffdd7700b8404231b37ec9c062\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.944348 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:01.944277 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:01.960713 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.960685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39cf6f2e19ecbd85104d624d7c507309-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal\" (UID: \"39cf6f2e19ecbd85104d624d7c507309\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.960793 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.960718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39cf6f2e19ecbd85104d624d7c507309-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal\" (UID: \"39cf6f2e19ecbd85104d624d7c507309\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.960793 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.960733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/196243ffdd7700b8404231b37ec9c062-config\") pod \"kube-apiserver-proxy-ip-10-0-139-196.ec2.internal\" (UID: \"196243ffdd7700b8404231b37ec9c062\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.960793 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.960761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/196243ffdd7700b8404231b37ec9c062-config\") pod \"kube-apiserver-proxy-ip-10-0-139-196.ec2.internal\" (UID: \"196243ffdd7700b8404231b37ec9c062\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.960793 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.960784 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39cf6f2e19ecbd85104d624d7c507309-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal\" (UID: \"39cf6f2e19ecbd85104d624d7c507309\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:01.960937 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:01.960801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39cf6f2e19ecbd85104d624d7c507309-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal\" (UID: \"39cf6f2e19ecbd85104d624d7c507309\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:02.044867 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.044836 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.064348 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.064321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:02.070237 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.070221 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" Jul 10 22:43:02.145831 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.145794 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.246374 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.246306 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.346905 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.346870 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.396412 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.396380 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 22:43:02.396929 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.396521 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 22:43:02.447851 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.447821 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.484006 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.483958 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 22:38:01 +0000 UTC" deadline="2028-01-04 14:03:06.111253616 +0000 UTC" Jul 10 22:43:02.484006 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.484002 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13023h20m3.627256224s" Jul 10 22:43:02.548296 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.548238 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.585605 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:02.585577 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196243ffdd7700b8404231b37ec9c062.slice/crio-3d2313102c02c179bf8b7daa6b1baee0cf6a547ce06725e8e1f1c7aa3714642b WatchSource:0}: Error finding container 3d2313102c02c179bf8b7daa6b1baee0cf6a547ce06725e8e1f1c7aa3714642b: Status 404 returned error can't find the container with id 3d2313102c02c179bf8b7daa6b1baee0cf6a547ce06725e8e1f1c7aa3714642b Jul 10 22:43:02.586160 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:02.586145 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cf6f2e19ecbd85104d624d7c507309.slice/crio-763425264c6f2a7464accc558dadf1c294f6d60b854d94933766418df5a31877 WatchSource:0}: Error finding container 763425264c6f2a7464accc558dadf1c294f6d60b854d94933766418df5a31877: Status 404 returned error can't find the container with id 763425264c6f2a7464accc558dadf1c294f6d60b854d94933766418df5a31877 Jul 10 22:43:02.591360 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.591345 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:43:02.613620 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.612387 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:43:02.631849 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.631801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" event={"ID":"39cf6f2e19ecbd85104d624d7c507309","Type":"ContainerStarted","Data":"763425264c6f2a7464accc558dadf1c294f6d60b854d94933766418df5a31877"} Jul 10 22:43:02.632775 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.632754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" event={"ID":"196243ffdd7700b8404231b37ec9c062","Type":"ContainerStarted","Data":"3d2313102c02c179bf8b7daa6b1baee0cf6a547ce06725e8e1f1c7aa3714642b"} Jul 10 22:43:02.648914 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.648893 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.749418 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:02.749388 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-196.ec2.internal\" not found" Jul 10 22:43:02.776480 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.776455 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:43:02.857903 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.857840 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" Jul 10 22:43:02.868510 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.868489 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 22:43:02.870196 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.870184 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" Jul 10 22:43:02.884781 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.884764 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 22:43:02.966701 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:02.966672 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:43:03.440253 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.440222 2577 apiserver.go:52] "Watching apiserver" Jul 10 22:43:03.449401 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.449374 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 22:43:03.451833 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.451804 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-t4hfp","openshift-network-diagnostics/network-check-target-b85vc","openshift-ovn-kubernetes/ovnkube-node-hmgsp","kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal","openshift-dns/node-resolver-ffv7g","openshift-image-registry/node-ca-5f5gr","openshift-multus/multus-gb4pk","openshift-multus/network-metrics-daemon-9hnxs","openshift-network-operator/iptables-alerter-tvstq","kube-system/global-pull-secret-syncer-n2bws","kube-system/konnectivity-agent-tg4cx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf","openshift-cluster-node-tuning-operator/tuned-wkw67","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal"] Jul 10 22:43:03.454463 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.454441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.454568 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.454529 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:03.456549 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.456528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.457302 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.457277 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 22:43:03.459416 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.459068 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jj2hk\"" Jul 10 22:43:03.459416 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.459132 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 22:43:03.459416 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.459252 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:43:03.459416 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.459358 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.461318 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.461298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:03.461409 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.461389 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:03.462559 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.462413 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 22:43:03.462559 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.462451 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 22:43:03.462559 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.462531 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 22:43:03.462559 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.462555 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 22:43:03.462854 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.462411 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2czwh\"" Jul 10 22:43:03.462854 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.462636 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 22:43:03.463473 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.463447 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.463870 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.463834 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 22:43:03.465650 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.465632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.466016 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.465715 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 22:43:03.466016 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.465892 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 22:43:03.466016 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.465740 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m562j\"" Jul 10 22:43:03.467862 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.467840 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 22:43:03.468111 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vch\" (UniqueName: \"kubernetes.io/projected/45d1c48a-a943-42eb-81b0-285c0761ea9c-kube-api-access-k5vch\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.468207 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c4d8088-b537-4bfe-9801-6d6cc21b0939-kubelet-config\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.468207 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysctl-d\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.468207 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.468207 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-sys\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-slash\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-etc-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlmk\" (UniqueName: \"kubernetes.io/projected/bc260c3c-875f-486d-b49f-2b615b3a4bc0-kube-api-access-hwlmk\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-var-lib-kubelet\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-kubelet\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-systemd-units\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovnkube-config\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-run\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c4d8088-b537-4bfe-9801-6d6cc21b0939-dbus\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468453 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 22:43:03.468478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468471 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-var-lib-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-ovn\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468607 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nq64j\"" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysconfig\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.468664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-node-log\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovnkube-script-lib\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-systemd\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.469107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d1c48a-a943-42eb-81b0-285c0761ea9c-tmp\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.469619 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-log-socket\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469619 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.469619 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-cni-bin\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469619 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-env-overrides\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469619 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-lib-modules\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.469619 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-run-netns\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469908 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469672 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-systemd\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469908 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-cni-netd\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469908 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovn-node-metrics-cert\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.469908 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-modprobe-d\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.469908 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-kubernetes\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.470152 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.469854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysctl-conf\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.470152 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.470061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-host\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.470152 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.470095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-tuned\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.471742 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.471682 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 22:43:03.471742 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.471690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.471884 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.471760 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9r2jp\"" Jul 10 22:43:03.471996 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.471692 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 22:43:03.472718 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.472545 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:43:03.472718 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.472562 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 22:43:03.474215 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.474179 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:03.474318 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.474292 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:03.476033 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.476013 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 22:43:03.476185 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.476167 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 22:43:03.476298 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.476279 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 22:43:03.476385 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.476298 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 22:43:03.476385 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.476353 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vg8hw\"" Jul 10 22:43:03.476599 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.476577 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.478898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.478877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.479251 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.479194 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 22:43:03.479251 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.479218 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hgfnw\"" Jul 10 22:43:03.479491 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.479473 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 22:43:03.481125 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.481104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.481635 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.481615 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 22:43:03.481635 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.481690 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zwrwq\"" Jul 10 22:43:03.481850 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.481789 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 22:43:03.483593 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.483578 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 22:43:03.483728 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.483589 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 22:43:03.483803 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.483675 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sx8xf\"" Jul 10 22:43:03.483887 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.483682 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 22:43:03.495299 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.495277 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jpzwk" Jul 10 22:43:03.502860 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.502840 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jpzwk" Jul 10 22:43:03.559133 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.559103 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 22:43:03.570344 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-daemon-config\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.570496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-systemd\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.570496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d1c48a-a943-42eb-81b0-285c0761ea9c-tmp\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.570496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-log-socket\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-env-overrides\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4edaf4b6-1410-4066-a868-955491cf08fd-konnectivity-ca\") pod \"konnectivity-agent-tg4cx\" (UID: \"4edaf4b6-1410-4066-a868-955491cf08fd\") " pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.570496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhlz\" (UniqueName: \"kubernetes.io/projected/5c654044-de95-4939-89e1-92b2618e6e56-kube-api-access-qrhlz\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-systemd\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-run-netns\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-run-netns\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-cni-netd\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570584 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-cni-netd\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-system-cni-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-log-socket\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-cni-bin\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-cni-multus\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-kubernetes\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c4d8088-b537-4bfe-9801-6d6cc21b0939-kubelet-config\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.570768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570766 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c0dea85-e823-40f2-a8ef-9430792b45c6-iptables-alerter-script\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-kubernetes\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-device-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570857 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c4d8088-b537-4bfe-9801-6d6cc21b0939-kubelet-config\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-sys\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-etc-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlmk\" (UniqueName: \"kubernetes.io/projected/bc260c3c-875f-486d-b49f-2b615b3a4bc0-kube-api-access-hwlmk\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-sys\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.570958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-os-release\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-netns\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-etc-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-var-lib-kubelet\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-env-overrides\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-kubelet\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovnkube-config\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.571386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xhq\" (UniqueName: \"kubernetes.io/projected/0c0dea85-e823-40f2-a8ef-9430792b45c6-kube-api-access-x8xhq\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-kubelet\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571126 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-multus-certs\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdk4c\" (UniqueName: \"kubernetes.io/projected/bd4ef950-777f-4f6a-9bd4-d08710af09b8-kube-api-access-vdk4c\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571223 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-var-lib-kubelet\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-run\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c4d8088-b537-4bfe-9801-6d6cc21b0939-dbus\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571310 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-run\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-var-lib-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571347 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c0dea85-e823-40f2-a8ef-9430792b45c6-host-slash\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571349 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-var-lib-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-cnibin\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c4d8088-b537-4bfe-9801-6d6cc21b0939-dbus\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-kubelet\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.571441 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:03.572140 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-conf-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.571516 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret podName:8c4d8088-b537-4bfe-9801-6d6cc21b0939 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:04.071477809 +0000 UTC m=+2.949169421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret") pod "global-pull-secret-syncer-n2bws" (UID: "8c4d8088-b537-4bfe-9801-6d6cc21b0939") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-node-log\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovnkube-script-lib\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/578887bd-bedf-4ef4-ad56-c7585e1573cb-hosts-file\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571597 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-openvswitch\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-k8s-cni-cncf-io\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovnkube-config\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-etc-kubernetes\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-node-log\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-cni-bin\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-lib-modules\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-systemd\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovn-node-metrics-cert\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.572836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-cni-bin\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-registration-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-systemd\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-modprobe-d\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571880 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-lib-modules\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysctl-conf\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-host\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-tuned\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.571960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vch\" (UniqueName: \"kubernetes.io/projected/45d1c48a-a943-42eb-81b0-285c0761ea9c-kube-api-access-k5vch\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysctl-conf\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572067 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-modprobe-d\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572092 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-host\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a046f2e1-b229-43a0-b693-1609f19e59dc-host\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovnkube-script-lib\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a046f2e1-b229-43a0-b693-1609f19e59dc-serviceca\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.573644 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysctl-d\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd4ef950-777f-4f6a-9bd4-d08710af09b8-cni-binary-copy\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4edaf4b6-1410-4066-a868-955491cf08fd-agent-certs\") pod \"konnectivity-agent-tg4cx\" (UID: \"4edaf4b6-1410-4066-a868-955491cf08fd\") " pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6k4c\" (UniqueName: \"kubernetes.io/projected/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-kube-api-access-h6k4c\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-socket-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqvr\" (UniqueName: \"kubernetes.io/projected/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-kube-api-access-cdqvr\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysctl-d\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-slash\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/578887bd-bedf-4ef4-ad56-c7585e1573cb-tmp-dir\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5k7\" (UniqueName: \"kubernetes.io/projected/578887bd-bedf-4ef4-ad56-c7585e1573cb-kube-api-access-fm5k7\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572881 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-slash\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-cni-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-socket-dir-parent\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.572981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-systemd-units\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-hostroot\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-systemd-units\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.574462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75t7\" (UniqueName: \"kubernetes.io/projected/a046f2e1-b229-43a0-b693-1609f19e59dc-kube-api-access-l75t7\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-os-release\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-sys-fs\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-ovn\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-system-cni-dir\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-run-ovn\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-cnibin\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysconfig\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573341 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc260c3c-875f-486d-b49f-2b615b3a4bc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.573436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-sysconfig\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.574553 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d1c48a-a943-42eb-81b0-285c0761ea9c-tmp\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.574587 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/45d1c48a-a943-42eb-81b0-285c0761ea9c-etc-tuned\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.575305 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.575085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc260c3c-875f-486d-b49f-2b615b3a4bc0-ovn-node-metrics-cert\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.582016 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.581995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlmk\" (UniqueName: \"kubernetes.io/projected/bc260c3c-875f-486d-b49f-2b615b3a4bc0-kube-api-access-hwlmk\") pod \"ovnkube-node-hmgsp\" (UID: \"bc260c3c-875f-486d-b49f-2b615b3a4bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.582325 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.582309 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vch\" (UniqueName: \"kubernetes.io/projected/45d1c48a-a943-42eb-81b0-285c0761ea9c-kube-api-access-k5vch\") pod \"tuned-wkw67\" (UID: \"45d1c48a-a943-42eb-81b0-285c0761ea9c\") " pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.673832 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-system-cni-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.673832 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-cni-bin\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-cni-multus\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-system-cni-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-cni-bin\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c0dea85-e823-40f2-a8ef-9430792b45c6-iptables-alerter-script\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-cni-multus\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.673999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-device-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-os-release\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-netns\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xhq\" (UniqueName: \"kubernetes.io/projected/0c0dea85-e823-40f2-a8ef-9430792b45c6-kube-api-access-x8xhq\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-multus-certs\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674126 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-device-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdk4c\" (UniqueName: \"kubernetes.io/projected/bd4ef950-777f-4f6a-9bd4-d08710af09b8-kube-api-access-vdk4c\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674177 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c0dea85-e823-40f2-a8ef-9430792b45c6-host-slash\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-multus-certs\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674248 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-os-release\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-netns\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-cnibin\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-kubelet\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674353 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c0dea85-e823-40f2-a8ef-9430792b45c6-host-slash\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-conf-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674381 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-cnibin\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-var-lib-kubelet\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/578887bd-bedf-4ef4-ad56-c7585e1573cb-hosts-file\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674423 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-conf-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-k8s-cni-cncf-io\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.674575 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-etc-kubernetes\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c0dea85-e823-40f2-a8ef-9430792b45c6-iptables-alerter-script\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-host-run-k8s-cni-cncf-io\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-registration-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674560 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/578887bd-bedf-4ef4-ad56-c7585e1573cb-hosts-file\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-etc-kubernetes\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a046f2e1-b229-43a0-b693-1609f19e59dc-host\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a046f2e1-b229-43a0-b693-1609f19e59dc-serviceca\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674622 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-registration-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.674646 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd4ef950-777f-4f6a-9bd4-d08710af09b8-cni-binary-copy\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a046f2e1-b229-43a0-b693-1609f19e59dc-host\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4edaf4b6-1410-4066-a868-955491cf08fd-agent-certs\") pod \"konnectivity-agent-tg4cx\" (UID: \"4edaf4b6-1410-4066-a868-955491cf08fd\") " pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.675424 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.674702 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs podName:c79360d2-84bf-4ebf-b7bc-e1e818e43cb0 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:04.174682552 +0000 UTC m=+3.052374149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs") pod "network-metrics-daemon-9hnxs" (UID: "c79360d2-84bf-4ebf-b7bc-e1e818e43cb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6k4c\" (UniqueName: \"kubernetes.io/projected/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-kube-api-access-h6k4c\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-socket-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqvr\" (UniqueName: \"kubernetes.io/projected/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-kube-api-access-cdqvr\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/578887bd-bedf-4ef4-ad56-c7585e1573cb-tmp-dir\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5k7\" (UniqueName: \"kubernetes.io/projected/578887bd-bedf-4ef4-ad56-c7585e1573cb-kube-api-access-fm5k7\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-cni-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-socket-dir-parent\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.674994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-socket-dir\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-hostroot\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75t7\" (UniqueName: \"kubernetes.io/projected/a046f2e1-b229-43a0-b693-1609f19e59dc-kube-api-access-l75t7\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-os-release\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-sys-fs\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.676183 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-cni-dir\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-system-cni-dir\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-cnibin\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-daemon-config\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd4ef950-777f-4f6a-9bd4-d08710af09b8-cni-binary-copy\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4edaf4b6-1410-4066-a868-955491cf08fd-konnectivity-ca\") pod \"konnectivity-agent-tg4cx\" (UID: \"4edaf4b6-1410-4066-a868-955491cf08fd\") " pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-socket-dir-parent\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhlz\" (UniqueName: \"kubernetes.io/projected/5c654044-de95-4939-89e1-92b2618e6e56-kube-api-access-qrhlz\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675339 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-system-cni-dir\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-cnibin\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd4ef950-777f-4f6a-9bd4-d08710af09b8-hostroot\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c654044-de95-4939-89e1-92b2618e6e56-os-release\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a046f2e1-b229-43a0-b693-1609f19e59dc-serviceca\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/578887bd-bedf-4ef4-ad56-c7585e1573cb-tmp-dir\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675499 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-sys-fs\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.675935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4edaf4b6-1410-4066-a868-955491cf08fd-konnectivity-ca\") pod \"konnectivity-agent-tg4cx\" (UID: \"4edaf4b6-1410-4066-a868-955491cf08fd\") " pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.676750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.676011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.677466 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.676062 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c654044-de95-4939-89e1-92b2618e6e56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.677466 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.676102 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd4ef950-777f-4f6a-9bd4-d08710af09b8-multus-daemon-config\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.677466 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.677251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4edaf4b6-1410-4066-a868-955491cf08fd-agent-certs\") pod \"konnectivity-agent-tg4cx\" (UID: \"4edaf4b6-1410-4066-a868-955491cf08fd\") " pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.683798 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.683557 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:43:03.683798 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.683579 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:43:03.683798 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.683594 2577 projected.go:196] Error preparing data for projected volume kube-api-access-j8q4z for pod openshift-network-diagnostics/network-check-target-b85vc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:03.683798 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:03.683654 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z podName:fba36b3d-f496-4988-b805-2ae5dd458684 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:04.183635911 +0000 UTC m=+3.061327530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j8q4z" (UniqueName: "kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z") pod "network-check-target-b85vc" (UID: "fba36b3d-f496-4988-b805-2ae5dd458684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:03.683798 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.683699 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdk4c\" (UniqueName: \"kubernetes.io/projected/bd4ef950-777f-4f6a-9bd4-d08710af09b8-kube-api-access-vdk4c\") pod \"multus-gb4pk\" (UID: \"bd4ef950-777f-4f6a-9bd4-d08710af09b8\") " pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.684309 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.684287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xhq\" (UniqueName: \"kubernetes.io/projected/0c0dea85-e823-40f2-a8ef-9430792b45c6-kube-api-access-x8xhq\") pod \"iptables-alerter-tvstq\" (UID: \"0c0dea85-e823-40f2-a8ef-9430792b45c6\") " pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.684932 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.684900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75t7\" (UniqueName: \"kubernetes.io/projected/a046f2e1-b229-43a0-b693-1609f19e59dc-kube-api-access-l75t7\") pod \"node-ca-5f5gr\" (UID: \"a046f2e1-b229-43a0-b693-1609f19e59dc\") " pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.685219 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.685196 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5k7\" (UniqueName: \"kubernetes.io/projected/578887bd-bedf-4ef4-ad56-c7585e1573cb-kube-api-access-fm5k7\") pod \"node-resolver-ffv7g\" (UID: \"578887bd-bedf-4ef4-ad56-c7585e1573cb\") " pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.685397 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.685364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhlz\" (UniqueName: \"kubernetes.io/projected/5c654044-de95-4939-89e1-92b2618e6e56-kube-api-access-qrhlz\") pod \"multus-additional-cni-plugins-t4hfp\" (UID: \"5c654044-de95-4939-89e1-92b2618e6e56\") " pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.686323 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.686302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqvr\" (UniqueName: \"kubernetes.io/projected/10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a-kube-api-access-cdqvr\") pod \"aws-ebs-csi-driver-node-8x5jf\" (UID: \"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:03.686684 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.686663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6k4c\" (UniqueName: \"kubernetes.io/projected/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-kube-api-access-h6k4c\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:03.723729 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.723647 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:43:03.769115 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.769081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wkw67" Jul 10 22:43:03.780111 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.780083 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:03.789931 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.789910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ffv7g" Jul 10 22:43:03.797630 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.797611 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5f5gr" Jul 10 22:43:03.806226 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.806207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvstq" Jul 10 22:43:03.813910 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.813890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gb4pk" Jul 10 22:43:03.821592 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.821576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" Jul 10 22:43:03.828246 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.828228 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:03.834822 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:03.834803 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" Jul 10 22:43:04.077829 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.077745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:04.077983 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.077911 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:04.078042 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.077994 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret podName:8c4d8088-b537-4bfe-9801-6d6cc21b0939 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:05.077963601 +0000 UTC m=+3.955655202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret") pod "global-pull-secret-syncer-n2bws" (UID: "8c4d8088-b537-4bfe-9801-6d6cc21b0939") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:04.178901 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.178870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:04.179096 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.179038 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:04.179144 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.179098 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs podName:c79360d2-84bf-4ebf-b7bc-e1e818e43cb0 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:05.17908248 +0000 UTC m=+4.056774077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs") pod "network-metrics-daemon-9hnxs" (UID: "c79360d2-84bf-4ebf-b7bc-e1e818e43cb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:04.279274 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.279235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:04.279466 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.279413 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:43:04.279466 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.279437 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:43:04.279466 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.279465 2577 projected.go:196] Error preparing data for projected volume kube-api-access-j8q4z for pod openshift-network-diagnostics/network-check-target-b85vc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:04.279628 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:04.279523 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z podName:fba36b3d-f496-4988-b805-2ae5dd458684 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:05.279504064 +0000 UTC m=+4.157195677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j8q4z" (UniqueName: "kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z") pod "network-check-target-b85vc" (UID: "fba36b3d-f496-4988-b805-2ae5dd458684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:04.468790 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:04.468759 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda046f2e1_b229_43a0_b693_1609f19e59dc.slice/crio-2d85f6dbfb2acc8d36c3ac93477ec7f98b5e625f3fb067fc0874b800776da53c WatchSource:0}: Error finding container 2d85f6dbfb2acc8d36c3ac93477ec7f98b5e625f3fb067fc0874b800776da53c: Status 404 returned error can't find the container with id 2d85f6dbfb2acc8d36c3ac93477ec7f98b5e625f3fb067fc0874b800776da53c Jul 10 22:43:04.469952 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:04.469924 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d1c48a_a943_42eb_81b0_285c0761ea9c.slice/crio-53ccbcc920349bacfc437736446efb3bd9c98928177252fcfaa845a659ce4c8f WatchSource:0}: Error finding container 53ccbcc920349bacfc437736446efb3bd9c98928177252fcfaa845a659ce4c8f: Status 404 returned error can't find the container with id 53ccbcc920349bacfc437736446efb3bd9c98928177252fcfaa845a659ce4c8f Jul 10 22:43:04.474724 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:04.474698 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0dea85_e823_40f2_a8ef_9430792b45c6.slice/crio-23ad7d75199662bb6f4a32c74ba7f57300b5a3e177a9bd76a6f816ee9a4da764 WatchSource:0}: Error finding container 23ad7d75199662bb6f4a32c74ba7f57300b5a3e177a9bd76a6f816ee9a4da764: Status 404 returned error can't find the container with id 23ad7d75199662bb6f4a32c74ba7f57300b5a3e177a9bd76a6f816ee9a4da764 Jul 10 22:43:04.475992 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:04.475923 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10afcfa9_a3d7_4bc9_877a_e6ec81c15e3a.slice/crio-63b5bc38a1af15c8ca0fd8ac5470fe572be069b8024b62d79120f21d1f672ca8 WatchSource:0}: Error finding container 63b5bc38a1af15c8ca0fd8ac5470fe572be069b8024b62d79120f21d1f672ca8: Status 404 returned error can't find the container with id 63b5bc38a1af15c8ca0fd8ac5470fe572be069b8024b62d79120f21d1f672ca8 Jul 10 22:43:04.477420 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:04.477304 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c654044_de95_4939_89e1_92b2618e6e56.slice/crio-df3b59dfa7722c8b82dfe159c763072acd06a29129162b1858cc552d57f4086a WatchSource:0}: Error finding container df3b59dfa7722c8b82dfe159c763072acd06a29129162b1858cc552d57f4086a: Status 404 returned error can't find the container with id df3b59dfa7722c8b82dfe159c763072acd06a29129162b1858cc552d57f4086a Jul 10 22:43:04.477962 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:04.477697 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edaf4b6_1410_4066_a868_955491cf08fd.slice/crio-2a4253a50624b3776c8766c20f54df1ad3b61efb8a3303cc606f8935a7ab007e WatchSource:0}: Error finding container 2a4253a50624b3776c8766c20f54df1ad3b61efb8a3303cc606f8935a7ab007e: Status 404 returned error can't find the container with id 2a4253a50624b3776c8766c20f54df1ad3b61efb8a3303cc606f8935a7ab007e Jul 10 22:43:04.506058 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.506001 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 22:38:03 +0000 UTC" deadline="2028-01-20 20:04:26.105359737 +0000 UTC" Jul 10 22:43:04.506058 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.506057 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13413h21m21.599306552s" Jul 10 22:43:04.637545 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.637295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" event={"ID":"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a","Type":"ContainerStarted","Data":"63b5bc38a1af15c8ca0fd8ac5470fe572be069b8024b62d79120f21d1f672ca8"} Jul 10 22:43:04.638456 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.638413 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gb4pk" event={"ID":"bd4ef950-777f-4f6a-9bd4-d08710af09b8","Type":"ContainerStarted","Data":"54026155b3071e6ee293c237ff7c5b3936e7436df49ff1f2c39f70e96cecc2a7"} Jul 10 22:43:04.639469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.639441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ffv7g" event={"ID":"578887bd-bedf-4ef4-ad56-c7585e1573cb","Type":"ContainerStarted","Data":"45d9b354e58b678e182c57e05a1ddb706b9ecf2fb121f07cb7557743dff02876"} Jul 10 22:43:04.640443 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.640423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"4925ee18ed7af22a11d89caa6dfc12ff47a3fbd843530689e0f564c65073f0fd"} Jul 10 22:43:04.641347 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.641324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tg4cx" event={"ID":"4edaf4b6-1410-4066-a868-955491cf08fd","Type":"ContainerStarted","Data":"2a4253a50624b3776c8766c20f54df1ad3b61efb8a3303cc606f8935a7ab007e"} Jul 10 22:43:04.642387 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.642363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvstq" event={"ID":"0c0dea85-e823-40f2-a8ef-9430792b45c6","Type":"ContainerStarted","Data":"23ad7d75199662bb6f4a32c74ba7f57300b5a3e177a9bd76a6f816ee9a4da764"} Jul 10 22:43:04.643368 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.643350 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5f5gr" event={"ID":"a046f2e1-b229-43a0-b693-1609f19e59dc","Type":"ContainerStarted","Data":"2d85f6dbfb2acc8d36c3ac93477ec7f98b5e625f3fb067fc0874b800776da53c"} Jul 10 22:43:04.645091 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.645070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" event={"ID":"196243ffdd7700b8404231b37ec9c062","Type":"ContainerStarted","Data":"ec866863393f34c71e0a9caeae0526301ea4faeafb1855d98a69f0a3ed8d8c49"} Jul 10 22:43:04.646282 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.646259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerStarted","Data":"df3b59dfa7722c8b82dfe159c763072acd06a29129162b1858cc552d57f4086a"} Jul 10 22:43:04.647312 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.647231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wkw67" event={"ID":"45d1c48a-a943-42eb-81b0-285c0761ea9c","Type":"ContainerStarted","Data":"53ccbcc920349bacfc437736446efb3bd9c98928177252fcfaa845a659ce4c8f"} Jul 10 22:43:04.659480 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:04.659426 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-196.ec2.internal" podStartSLOduration=2.6594060710000003 podStartE2EDuration="2.659406071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:04.658714486 +0000 UTC m=+3.536406107" watchObservedRunningTime="2026-07-10 22:43:04.659406071 +0000 UTC m=+3.537097693" Jul 10 22:43:05.087986 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.087292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:05.087986 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.087462 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:05.087986 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.087525 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret podName:8c4d8088-b537-4bfe-9801-6d6cc21b0939 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:07.087507518 +0000 UTC m=+5.965199119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret") pod "global-pull-secret-syncer-n2bws" (UID: "8c4d8088-b537-4bfe-9801-6d6cc21b0939") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:05.188164 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.188125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:05.188360 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.188295 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:05.188360 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.188359 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs podName:c79360d2-84bf-4ebf-b7bc-e1e818e43cb0 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:07.188338818 +0000 UTC m=+6.066030415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs") pod "network-metrics-daemon-9hnxs" (UID: "c79360d2-84bf-4ebf-b7bc-e1e818e43cb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:05.289621 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.288987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:05.289621 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.289166 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:43:05.289621 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.289186 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:43:05.289621 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.289199 2577 projected.go:196] Error preparing data for projected volume kube-api-access-j8q4z for pod openshift-network-diagnostics/network-check-target-b85vc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:05.289621 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.289255 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z podName:fba36b3d-f496-4988-b805-2ae5dd458684 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:07.289237521 +0000 UTC m=+6.166929121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j8q4z" (UniqueName: "kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z") pod "network-check-target-b85vc" (UID: "fba36b3d-f496-4988-b805-2ae5dd458684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:05.506539 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.506373 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 22:38:03 +0000 UTC" deadline="2028-01-30 02:01:43.040341067 +0000 UTC" Jul 10 22:43:05.506539 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.506414 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13635h18m37.533931164s" Jul 10 22:43:05.632776 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.632747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:05.632928 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.632874 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:05.633293 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.633276 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:05.633403 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.633383 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:05.633462 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.633443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:05.633558 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:05.633509 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:05.667445 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.667406 2577 generic.go:356] "Generic (PLEG): container finished" podID="39cf6f2e19ecbd85104d624d7c507309" containerID="6e633a5c9ec821d59da71cf07f348fd71af7916ca4bd5f820b98083699ceb48b" exitCode=0 Jul 10 22:43:05.667985 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:05.667941 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" event={"ID":"39cf6f2e19ecbd85104d624d7c507309","Type":"ContainerDied","Data":"6e633a5c9ec821d59da71cf07f348fd71af7916ca4bd5f820b98083699ceb48b"} Jul 10 22:43:06.700998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:06.700818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" event={"ID":"39cf6f2e19ecbd85104d624d7c507309","Type":"ContainerStarted","Data":"1fb14fb804f76aa8db54c8daf0c9b6720a02e8b55df854d62a34c1b71d897897"} Jul 10 22:43:07.104037 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:07.103918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:07.104208 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.104114 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:07.104208 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.104173 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret podName:8c4d8088-b537-4bfe-9801-6d6cc21b0939 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:11.104155865 +0000 UTC m=+9.981847464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret") pod "global-pull-secret-syncer-n2bws" (UID: "8c4d8088-b537-4bfe-9801-6d6cc21b0939") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:07.204764 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:07.204731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:07.204953 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.204891 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:07.205039 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.204959 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs podName:c79360d2-84bf-4ebf-b7bc-e1e818e43cb0 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:11.204939443 +0000 UTC m=+10.082631054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs") pod "network-metrics-daemon-9hnxs" (UID: "c79360d2-84bf-4ebf-b7bc-e1e818e43cb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:07.305485 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:07.305452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:07.305662 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.305649 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:43:07.305727 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.305674 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:43:07.305727 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.305687 2577 projected.go:196] Error preparing data for projected volume kube-api-access-j8q4z for pod openshift-network-diagnostics/network-check-target-b85vc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:07.305828 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.305742 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z podName:fba36b3d-f496-4988-b805-2ae5dd458684 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:11.305722311 +0000 UTC m=+10.183413927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j8q4z" (UniqueName: "kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z") pod "network-check-target-b85vc" (UID: "fba36b3d-f496-4988-b805-2ae5dd458684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:07.630678 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:07.630125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:07.630678 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.630252 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:07.630678 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:07.630675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:07.631012 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:07.630702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:07.631012 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.630769 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:07.631012 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:07.630861 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:09.630255 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:09.630223 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:09.630680 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:09.630352 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:09.630680 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:09.630477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:09.630680 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:09.630602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:09.630680 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:09.630602 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:09.630895 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:09.630683 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:11.140233 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:11.140170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:11.140682 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.140351 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:11.140682 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.140429 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret podName:8c4d8088-b537-4bfe-9801-6d6cc21b0939 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:19.140409182 +0000 UTC m=+18.018100794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret") pod "global-pull-secret-syncer-n2bws" (UID: "8c4d8088-b537-4bfe-9801-6d6cc21b0939") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:11.241157 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:11.240691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:11.241157 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.240900 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:11.241157 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.240960 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs podName:c79360d2-84bf-4ebf-b7bc-e1e818e43cb0 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:19.240942674 +0000 UTC m=+18.118634277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs") pod "network-metrics-daemon-9hnxs" (UID: "c79360d2-84bf-4ebf-b7bc-e1e818e43cb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:11.342041 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:11.341996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:11.342222 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.342181 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:43:11.342222 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.342206 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:43:11.342222 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.342220 2577 projected.go:196] Error preparing data for projected volume kube-api-access-j8q4z for pod openshift-network-diagnostics/network-check-target-b85vc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:11.342392 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.342287 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z podName:fba36b3d-f496-4988-b805-2ae5dd458684 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:19.34226801 +0000 UTC m=+18.219959620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j8q4z" (UniqueName: "kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z") pod "network-check-target-b85vc" (UID: "fba36b3d-f496-4988-b805-2ae5dd458684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:11.632296 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:11.632219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:11.632445 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.632405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:11.632879 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:11.632856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:11.633040 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.633007 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:11.633101 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:11.633048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:11.633165 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:11.633145 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:13.630036 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:13.630003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:13.630036 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:13.630030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:13.630036 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:13.630044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:13.630594 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:13.630143 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:13.630594 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:13.630217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:13.630594 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:13.630309 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:15.629516 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:15.629484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:15.630042 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:15.629485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:15.630042 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:15.629609 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:15.630042 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:15.629644 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:15.630042 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:15.629779 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:15.630042 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:15.629883 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:17.632560 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:17.632523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:17.633003 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:17.632632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:17.633003 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:17.632638 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:17.633003 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:17.632665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:17.633003 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:17.632772 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:17.633003 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:17.632857 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:19.203060 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:19.203025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:19.203467 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.203163 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:19.203467 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.203223 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret podName:8c4d8088-b537-4bfe-9801-6d6cc21b0939 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:35.203207306 +0000 UTC m=+34.080898908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret") pod "global-pull-secret-syncer-n2bws" (UID: "8c4d8088-b537-4bfe-9801-6d6cc21b0939") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:19.303776 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:19.303749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:19.303881 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.303868 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:19.303930 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.303918 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs podName:c79360d2-84bf-4ebf-b7bc-e1e818e43cb0 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:35.303906638 +0000 UTC m=+34.181598235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs") pod "network-metrics-daemon-9hnxs" (UID: "c79360d2-84bf-4ebf-b7bc-e1e818e43cb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:19.404353 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:19.404301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:19.404555 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.404496 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:43:19.404555 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.404522 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:43:19.404555 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.404534 2577 projected.go:196] Error preparing data for projected volume kube-api-access-j8q4z for pod openshift-network-diagnostics/network-check-target-b85vc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:19.404711 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.404598 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z podName:fba36b3d-f496-4988-b805-2ae5dd458684 nodeName:}" failed. No retries permitted until 2026-07-10 22:43:35.404579127 +0000 UTC m=+34.282270727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j8q4z" (UniqueName: "kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z") pod "network-check-target-b85vc" (UID: "fba36b3d-f496-4988-b805-2ae5dd458684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:19.633181 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:19.633103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:19.633181 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:19.633136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:19.633399 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:19.633105 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:19.633399 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.633245 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:19.633399 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.633310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:19.633399 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:19.633377 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:21.630751 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:21.630716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:21.631240 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:21.630814 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:21.631240 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:21.630823 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:21.631240 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:21.630938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:21.631240 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:21.631024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:21.631240 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:21.631097 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:23.630202 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.630043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:23.630859 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.630089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:23.630859 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:23.630289 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:23.630859 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:23.630380 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:23.630859 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.630116 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:23.630859 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:23.630503 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:23.736567 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.736537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" event={"ID":"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a","Type":"ContainerStarted","Data":"ca59930fb9034c355a894fd0c1359d38de1747561becc2890fb3566a23c4324d"} Jul 10 22:43:23.738154 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.738005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gb4pk" event={"ID":"bd4ef950-777f-4f6a-9bd4-d08710af09b8","Type":"ContainerStarted","Data":"c07479f197dba05d64212f7f3664e1f5e42a212358cf92b18c348042d8e69f56"} Jul 10 22:43:23.739402 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.739379 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ffv7g" event={"ID":"578887bd-bedf-4ef4-ad56-c7585e1573cb","Type":"ContainerStarted","Data":"2f7a51928de80ef980bb2f99cbf0252ce056d4c0fec60f21e08dc1e6fdeb6a25"} Jul 10 22:43:23.741638 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.741602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:43:23.742009 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.741967 2577 generic.go:356] "Generic (PLEG): container finished" podID="bc260c3c-875f-486d-b49f-2b615b3a4bc0" containerID="5dc41d6eea457b733d8e98a162a1c1d063314fee1a84dc35d39727b3f51fa04a" exitCode=1 Jul 10 22:43:23.742009 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.741999 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"5907b6839e24b22f7ef751fbbff0bad74add49f1722947aa4a208784f75f52f9"} Jul 10 22:43:23.742225 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.742050 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"42cdba5f841f89aed9324fb550d0a7d0451741ec499514eadbeb83bd40fdb1f8"} Jul 10 22:43:23.742225 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.742094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerDied","Data":"5dc41d6eea457b733d8e98a162a1c1d063314fee1a84dc35d39727b3f51fa04a"} Jul 10 22:43:23.742225 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.742142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"9d46ab2281ed08093e74260bc34aa86c2538a4c225c06eb946a6a058fada9a61"} Jul 10 22:43:23.743438 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.743408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tg4cx" event={"ID":"4edaf4b6-1410-4066-a868-955491cf08fd","Type":"ContainerStarted","Data":"0bc0f9662c2de36574f1dcc4e9a2e0584286c67df2d4d365a82a821007859478"} Jul 10 22:43:23.744806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.744788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5f5gr" event={"ID":"a046f2e1-b229-43a0-b693-1609f19e59dc","Type":"ContainerStarted","Data":"16d27e93c93818a974942b7c099bbf290bb6e25f8eed8259d57835a41f357125"} Jul 10 22:43:23.746517 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.746489 2577 generic.go:356] "Generic (PLEG): container finished" podID="5c654044-de95-4939-89e1-92b2618e6e56" containerID="7db3022cae4745b31bbd5e62d2867c232e8d75ed5250df9104cb945213ddffa0" exitCode=0 Jul 10 22:43:23.746613 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.746568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerDied","Data":"7db3022cae4745b31bbd5e62d2867c232e8d75ed5250df9104cb945213ddffa0"} Jul 10 22:43:23.748209 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.748097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wkw67" event={"ID":"45d1c48a-a943-42eb-81b0-285c0761ea9c","Type":"ContainerStarted","Data":"74fc95d8bc0f76b7bfec7d1f56e2537b83628a08bd9e5f491a5f4cd101fe478c"} Jul 10 22:43:23.754319 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.754286 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gb4pk" podStartSLOduration=3.920319091 podStartE2EDuration="22.754273972s" totalImagesPullingTime="18.833954881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:23.75413592 +0000 UTC m=+22.631827541" watchObservedRunningTime="2026-07-10 22:43:23.754273972 +0000 UTC m=+22.631965594" Jul 10 22:43:23.754632 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.754610 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-196.ec2.internal" podStartSLOduration=21.754603771 podStartE2EDuration="21.754603771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:06.717045875 +0000 UTC m=+5.594737521" watchObservedRunningTime="2026-07-10 22:43:23.754603771 +0000 UTC m=+22.632295389" Jul 10 22:43:23.774414 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.774377 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wkw67" podStartSLOduration=3.99303749 podStartE2EDuration="22.774365699s" totalImagesPullingTime="18.781328209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:23.774024712 +0000 UTC m=+22.651716352" watchObservedRunningTime="2026-07-10 22:43:23.774365699 +0000 UTC m=+22.652057318" Jul 10 22:43:23.788508 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.788473 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ffv7g" podStartSLOduration=4.098229869 podStartE2EDuration="22.78846049s" totalImagesPullingTime="18.690230621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:23.788152304 +0000 UTC m=+22.665843923" watchObservedRunningTime="2026-07-10 22:43:23.78846049 +0000 UTC m=+22.666152109" Jul 10 22:43:23.823243 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.823202 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tg4cx" podStartSLOduration=8.020462333 podStartE2EDuration="22.823187522s" totalImagesPullingTime="14.802725189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:23.822877965 +0000 UTC m=+22.700569614" watchObservedRunningTime="2026-07-10 22:43:23.823187522 +0000 UTC m=+22.700879141" Jul 10 22:43:23.837018 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:23.836959 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5f5gr" podStartSLOduration=4.133781984 podStartE2EDuration="22.836945313s" totalImagesPullingTime="18.703163329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:23.836073505 +0000 UTC m=+22.713765124" watchObservedRunningTime="2026-07-10 22:43:23.836945313 +0000 UTC m=+22.714636996" Jul 10 22:43:24.752576 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:24.752532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvstq" event={"ID":"0c0dea85-e823-40f2-a8ef-9430792b45c6","Type":"ContainerStarted","Data":"2fdad6375e24f26a0ef56aa57ad8f9e9ef5a09c0854405517e8d58811431ca48"} Jul 10 22:43:24.755894 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:24.755869 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:43:24.756385 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:24.756356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"96c10a96a863ac1fdd88c958f5c3bdb3660c551060fcb33138ddf932d09bf323"} Jul 10 22:43:24.756507 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:24.756438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"cfe99054459ddd55a498b00beee371e28a3ff547b89d963f032dd7ca231d82c6"} Jul 10 22:43:24.768262 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:24.768212 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tvstq" podStartSLOduration=5.069970982 podStartE2EDuration="23.768163614s" totalImagesPullingTime="18.698192632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:24.767385584 +0000 UTC m=+23.645077224" watchObservedRunningTime="2026-07-10 22:43:24.768163614 +0000 UTC m=+23.645855234" Jul 10 22:43:24.776521 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:24.776498 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 22:43:25.529529 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.529408 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T22:43:24.776515634Z","UUID":"2ace1e28-f88e-439a-8918-dba317adcb73","Handler":null,"Name":"","Endpoint":""} Jul 10 22:43:25.531327 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.531295 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 22:43:25.531458 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.531341 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 22:43:25.629798 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.629765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:25.629998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.629769 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:25.629998 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:25.629900 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:25.629998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.629912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:25.630181 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:25.630046 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:25.630181 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:25.630139 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:25.634277 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.634258 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:25.634844 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.634798 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:25.760172 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.760101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" event={"ID":"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a","Type":"ContainerStarted","Data":"aeb9ca8849d0528e915a6fbabbe1ebf9f25834a02a251be0f9e078462a906968"} Jul 10 22:43:25.760599 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.760413 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:25.760903 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:25.760883 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tg4cx" Jul 10 22:43:26.764915 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:26.764681 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" event={"ID":"10afcfa9-a3d7-4bc9-877a-e6ec81c15e3a","Type":"ContainerStarted","Data":"b813dc213e438d2f9c775856d4c763ad2eba275c5b6ec219af9ccd9c892e4dfa"} Jul 10 22:43:26.768021 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:26.767991 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:43:26.768447 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:26.768407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"036eff935948b315698286f7f887e20fad0a55cceadde97f2f0f28d6bc7c6ff5"} Jul 10 22:43:26.786065 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:26.786010 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8x5jf" podStartSLOduration=4.890396777 podStartE2EDuration="25.785990841s" totalImagesPullingTime="20.895594064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:26.785678313 +0000 UTC m=+25.663370149" watchObservedRunningTime="2026-07-10 22:43:26.785990841 +0000 UTC m=+25.663682455" Jul 10 22:43:27.629653 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:27.629606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:27.629885 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:27.629604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:27.629885 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:27.629724 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:27.629885 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:27.629743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:27.629885 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:27.629865 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:27.630159 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:27.629965 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:28.776224 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.776060 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:43:28.777086 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.776595 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"c6c7c0a5f7fc0beded0eb071a094b968576135bef3df84f68907b83dc0e31186"} Jul 10 22:43:28.777086 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.776952 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:28.777086 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.777040 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:28.777086 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.777084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:28.777286 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.777130 2577 scope.go:117] "RemoveContainer" containerID="5dc41d6eea457b733d8e98a162a1c1d063314fee1a84dc35d39727b3f51fa04a" Jul 10 22:43:28.778600 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.778575 2577 generic.go:356] "Generic (PLEG): container finished" podID="5c654044-de95-4939-89e1-92b2618e6e56" containerID="4230ed0d01519c4573ed2ce6c391a7847432765eb1c936434b5ef848c5d333c8" exitCode=0 Jul 10 22:43:28.778726 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.778612 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerDied","Data":"4230ed0d01519c4573ed2ce6c391a7847432765eb1c936434b5ef848c5d333c8"} Jul 10 22:43:28.793460 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.793437 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:28.793758 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:28.793731 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:43:29.629928 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:29.629847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:29.629928 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:29.629847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:29.630174 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:29.629963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:29.630174 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:29.629987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:29.630174 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:29.630102 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:29.630174 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:29.630158 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:29.784015 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:29.783985 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:43:29.784370 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:29.784341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" event={"ID":"bc260c3c-875f-486d-b49f-2b615b3a4bc0","Type":"ContainerStarted","Data":"6250060054bf81bb5b6d7f3be334c1c9c28c19f6d6aa5efb50f17019cd60290f"} Jul 10 22:43:29.816513 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:29.816458 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" podStartSLOduration=9.997979975 podStartE2EDuration="28.816442065s" totalImagesPullingTime="18.81846209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:29.814696595 +0000 UTC m=+28.692388214" watchObservedRunningTime="2026-07-10 22:43:29.816442065 +0000 UTC m=+28.694133684" Jul 10 22:43:30.788531 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:30.788493 2577 generic.go:356] "Generic (PLEG): container finished" podID="5c654044-de95-4939-89e1-92b2618e6e56" containerID="24e8589e8f93d9ab96dfe62db7cc2a7514ff2d4d53d51b5ecf3d10206f318fc5" exitCode=0 Jul 10 22:43:30.788955 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:30.788569 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerDied","Data":"24e8589e8f93d9ab96dfe62db7cc2a7514ff2d4d53d51b5ecf3d10206f318fc5"} Jul 10 22:43:31.456022 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:31.455827 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n2bws"] Jul 10 22:43:31.456222 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:31.456138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:31.456289 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:31.456249 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:31.458953 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:31.458908 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b85vc"] Jul 10 22:43:31.459103 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:31.459076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:31.459207 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:31.459184 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:31.459479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:31.459459 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9hnxs"] Jul 10 22:43:31.459552 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:31.459545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:31.459641 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:31.459621 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:32.797758 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:32.797717 2577 generic.go:356] "Generic (PLEG): container finished" podID="5c654044-de95-4939-89e1-92b2618e6e56" containerID="c64c6415e211c20f36a620874224dd247d195f3ac7ed548269fcb20c08f9db71" exitCode=0 Jul 10 22:43:32.798246 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:32.797784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerDied","Data":"c64c6415e211c20f36a620874224dd247d195f3ac7ed548269fcb20c08f9db71"} Jul 10 22:43:33.629922 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:33.629885 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:33.630128 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:33.629886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:33.630128 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:33.629886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:33.630128 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:33.630045 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:33.630331 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:33.630157 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:33.630331 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:33.630273 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:35.228563 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.228531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:35.229085 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.228664 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:35.229085 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.228724 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret podName:8c4d8088-b537-4bfe-9801-6d6cc21b0939 nodeName:}" failed. No retries permitted until 2026-07-10 22:44:07.228706176 +0000 UTC m=+66.106397784 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret") pod "global-pull-secret-syncer-n2bws" (UID: "8c4d8088-b537-4bfe-9801-6d6cc21b0939") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:43:35.329302 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.329265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:35.329487 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.329387 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:35.329487 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.329453 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs podName:c79360d2-84bf-4ebf-b7bc-e1e818e43cb0 nodeName:}" failed. No retries permitted until 2026-07-10 22:44:07.329437377 +0000 UTC m=+66.207128980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs") pod "network-metrics-daemon-9hnxs" (UID: "c79360d2-84bf-4ebf-b7bc-e1e818e43cb0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:43:35.430102 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.430063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:35.430280 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.430262 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:43:35.430365 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.430285 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:43:35.430365 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.430298 2577 projected.go:196] Error preparing data for projected volume kube-api-access-j8q4z for pod openshift-network-diagnostics/network-check-target-b85vc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:35.430463 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.430365 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z podName:fba36b3d-f496-4988-b805-2ae5dd458684 nodeName:}" failed. No retries permitted until 2026-07-10 22:44:07.430345229 +0000 UTC m=+66.308036840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j8q4z" (UniqueName: "kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z") pod "network-check-target-b85vc" (UID: "fba36b3d-f496-4988-b805-2ae5dd458684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:43:35.629832 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.629736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:35.629832 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.629774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:35.630073 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.629736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:35.630073 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.629890 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hnxs" podUID="c79360d2-84bf-4ebf-b7bc-e1e818e43cb0" Jul 10 22:43:35.630073 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.630016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n2bws" podUID="8c4d8088-b537-4bfe-9801-6d6cc21b0939" Jul 10 22:43:35.630190 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:35.630119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b85vc" podUID="fba36b3d-f496-4988-b805-2ae5dd458684" Jul 10 22:43:35.917992 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.917951 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-196.ec2.internal" event="NodeReady" Jul 10 22:43:35.918173 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.918120 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 22:43:35.962087 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.962054 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gqks5"] Jul 10 22:43:35.987843 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.987811 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-spnht"] Jul 10 22:43:35.988031 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.987944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:35.990965 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.990938 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4q74\"" Jul 10 22:43:35.991100 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.990938 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 22:43:35.991100 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:35.990944 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 22:43:36.011936 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.011903 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqks5"] Jul 10 22:43:36.012130 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.012010 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-spnht"] Jul 10 22:43:36.012130 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.012045 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-65vzt"] Jul 10 22:43:36.012130 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.012048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.015023 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.014821 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 22:43:36.015023 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.014867 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 22:43:36.015023 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.014880 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 22:43:36.015023 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.014935 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 22:43:36.015023 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.015004 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-spbmt\"" Jul 10 22:43:36.027891 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.027868 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-65vzt"] Jul 10 22:43:36.028024 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.027964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:36.031209 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.031155 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 22:43:36.031811 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.031685 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d7gcw\"" Jul 10 22:43:36.031959 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.031947 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 22:43:36.032056 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.031963 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 22:43:36.134573 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-crio-socket\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.134573 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-data-volume\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.134803 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvf66\" (UniqueName: \"kubernetes.io/projected/f6e50fe6-8a37-4770-9f2a-377db6af6cba-kube-api-access-xvf66\") pod \"ingress-canary-65vzt\" (UID: \"f6e50fe6-8a37-4770-9f2a-377db6af6cba\") " pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:36.134803 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6e50fe6-8a37-4770-9f2a-377db6af6cba-cert\") pod \"ingress-canary-65vzt\" (UID: \"f6e50fe6-8a37-4770-9f2a-377db6af6cba\") " pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:36.134803 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9jf\" (UniqueName: \"kubernetes.io/projected/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-kube-api-access-dg9jf\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.134803 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ad68902-2c53-433c-a64b-561a56cee8d4-metrics-tls\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.135004 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad68902-2c53-433c-a64b-561a56cee8d4-config-volume\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.135004 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7l78\" (UniqueName: \"kubernetes.io/projected/3ad68902-2c53-433c-a64b-561a56cee8d4-kube-api-access-s7l78\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.135004 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad68902-2c53-433c-a64b-561a56cee8d4-tmp-dir\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.135004 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.135004 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.134941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.236076 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.235991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-crio-socket\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.236076 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-data-volume\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvf66\" (UniqueName: \"kubernetes.io/projected/f6e50fe6-8a37-4770-9f2a-377db6af6cba-kube-api-access-xvf66\") pod \"ingress-canary-65vzt\" (UID: \"f6e50fe6-8a37-4770-9f2a-377db6af6cba\") " pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6e50fe6-8a37-4770-9f2a-377db6af6cba-cert\") pod \"ingress-canary-65vzt\" (UID: \"f6e50fe6-8a37-4770-9f2a-377db6af6cba\") " pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9jf\" (UniqueName: \"kubernetes.io/projected/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-kube-api-access-dg9jf\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ad68902-2c53-433c-a64b-561a56cee8d4-metrics-tls\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad68902-2c53-433c-a64b-561a56cee8d4-config-volume\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-crio-socket\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7l78\" (UniqueName: \"kubernetes.io/projected/3ad68902-2c53-433c-a64b-561a56cee8d4-kube-api-access-s7l78\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad68902-2c53-433c-a64b-561a56cee8d4-tmp-dir\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.236570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-data-volume\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.237160 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.236683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad68902-2c53-433c-a64b-561a56cee8d4-tmp-dir\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.237160 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.237079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad68902-2c53-433c-a64b-561a56cee8d4-config-volume\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.237160 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.237102 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.240688 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.240666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.240827 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.240741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ad68902-2c53-433c-a64b-561a56cee8d4-metrics-tls\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.240890 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.240830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6e50fe6-8a37-4770-9f2a-377db6af6cba-cert\") pod \"ingress-canary-65vzt\" (UID: \"f6e50fe6-8a37-4770-9f2a-377db6af6cba\") " pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:36.246604 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.246385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9jf\" (UniqueName: \"kubernetes.io/projected/f1988cf5-429e-4b86-8a35-0ee1ddabd8e9-kube-api-access-dg9jf\") pod \"insights-runtime-extractor-spnht\" (UID: \"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9\") " pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.246812 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.246787 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvf66\" (UniqueName: \"kubernetes.io/projected/f6e50fe6-8a37-4770-9f2a-377db6af6cba-kube-api-access-xvf66\") pod \"ingress-canary-65vzt\" (UID: \"f6e50fe6-8a37-4770-9f2a-377db6af6cba\") " pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:36.247013 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.246986 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7l78\" (UniqueName: \"kubernetes.io/projected/3ad68902-2c53-433c-a64b-561a56cee8d4-kube-api-access-s7l78\") pod \"dns-default-gqks5\" (UID: \"3ad68902-2c53-433c-a64b-561a56cee8d4\") " pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.300043 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.300005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:36.323166 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.323128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-spnht" Jul 10 22:43:36.337810 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:36.337780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-65vzt" Jul 10 22:43:37.003901 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.003869 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-65vzt"] Jul 10 22:43:37.011570 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.011542 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqks5"] Jul 10 22:43:37.012314 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.012300 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-spnht"] Jul 10 22:43:37.034077 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:37.034044 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e50fe6_8a37_4770_9f2a_377db6af6cba.slice/crio-93e795898a5392b008dba881bdc5db778f34ab2ad52929f4bf90ea858954a4d0 WatchSource:0}: Error finding container 93e795898a5392b008dba881bdc5db778f34ab2ad52929f4bf90ea858954a4d0: Status 404 returned error can't find the container with id 93e795898a5392b008dba881bdc5db778f34ab2ad52929f4bf90ea858954a4d0 Jul 10 22:43:37.034545 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:37.034516 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1988cf5_429e_4b86_8a35_0ee1ddabd8e9.slice/crio-a679732597b295c646d25cca5264afb17b53f685defb968c138b760ba6064363 WatchSource:0}: Error finding container a679732597b295c646d25cca5264afb17b53f685defb968c138b760ba6064363: Status 404 returned error can't find the container with id a679732597b295c646d25cca5264afb17b53f685defb968c138b760ba6064363 Jul 10 22:43:37.035409 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:37.035387 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad68902_2c53_433c_a64b_561a56cee8d4.slice/crio-9c1edb75184e03e3c7d41f9f6c1640fd82102a616db66bd9f5fdd1046b763c7b WatchSource:0}: Error finding container 9c1edb75184e03e3c7d41f9f6c1640fd82102a616db66bd9f5fdd1046b763c7b: Status 404 returned error can't find the container with id 9c1edb75184e03e3c7d41f9f6c1640fd82102a616db66bd9f5fdd1046b763c7b Jul 10 22:43:37.630035 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.630001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:43:37.630665 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.630001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:43:37.630665 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.630173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:43:37.635240 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.635216 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 22:43:37.635571 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.635542 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 22:43:37.635846 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.635828 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wwzq4\"" Jul 10 22:43:37.636059 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.636043 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 22:43:37.636142 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.636100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 22:43:37.636260 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.636243 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vx2kj\"" Jul 10 22:43:37.814101 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.814025 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqks5" event={"ID":"3ad68902-2c53-433c-a64b-561a56cee8d4","Type":"ContainerStarted","Data":"9c1edb75184e03e3c7d41f9f6c1640fd82102a616db66bd9f5fdd1046b763c7b"} Jul 10 22:43:37.816372 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.816283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spnht" event={"ID":"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9","Type":"ContainerStarted","Data":"a679732597b295c646d25cca5264afb17b53f685defb968c138b760ba6064363"} Jul 10 22:43:37.819373 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.819317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-65vzt" event={"ID":"f6e50fe6-8a37-4770-9f2a-377db6af6cba","Type":"ContainerStarted","Data":"93e795898a5392b008dba881bdc5db778f34ab2ad52929f4bf90ea858954a4d0"} Jul 10 22:43:37.824416 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.824371 2577 generic.go:356] "Generic (PLEG): container finished" podID="5c654044-de95-4939-89e1-92b2618e6e56" containerID="ef046d9d47f3b077eacca23a903cb71bebbcb7f8a74be5e86061fd8c239f2635" exitCode=0 Jul 10 22:43:37.824540 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:37.824436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerDied","Data":"ef046d9d47f3b077eacca23a903cb71bebbcb7f8a74be5e86061fd8c239f2635"} Jul 10 22:43:38.832013 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:38.831955 2577 generic.go:356] "Generic (PLEG): container finished" podID="5c654044-de95-4939-89e1-92b2618e6e56" containerID="3d06a5c4145ab8df5dc7349da412fdb9c4b9176ab318ba8314bc6b86052b030d" exitCode=0 Jul 10 22:43:38.832477 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:38.832036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerDied","Data":"3d06a5c4145ab8df5dc7349da412fdb9c4b9176ab318ba8314bc6b86052b030d"} Jul 10 22:43:43.846919 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.846610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-65vzt" event={"ID":"f6e50fe6-8a37-4770-9f2a-377db6af6cba","Type":"ContainerStarted","Data":"756f7c5e2702639677e8c6f7889f6188e84ca91efbee6772087b482c2fbef889"} Jul 10 22:43:43.849900 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.849877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" event={"ID":"5c654044-de95-4939-89e1-92b2618e6e56","Type":"ContainerStarted","Data":"584c40c7c3e5db5263c1d6b28c2e791b01848601361cd40096401b13893c7140"} Jul 10 22:43:43.851577 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.851547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqks5" event={"ID":"3ad68902-2c53-433c-a64b-561a56cee8d4","Type":"ContainerStarted","Data":"77347887e04695b6cbc15fe2ddd9f87e3c05387236d11ff7368c475271e98679"} Jul 10 22:43:43.851678 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.851587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqks5" event={"ID":"3ad68902-2c53-433c-a64b-561a56cee8d4","Type":"ContainerStarted","Data":"af99ae54d4061e2ee8fc9df2ba84ff51cae1d8c2728673c857140761f0acc7b1"} Jul 10 22:43:43.851678 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.851671 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:43.852899 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.852880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spnht" event={"ID":"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9","Type":"ContainerStarted","Data":"281fed42263d91de28a3e0871bf3f66e647ad1fd698f8d4aab2309b5117b5075"} Jul 10 22:43:43.891303 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.891250 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-65vzt" podStartSLOduration=3.003625436 podStartE2EDuration="8.891231012s" totalImagesPullingTime="5.887605576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:43.863571108 +0000 UTC m=+42.741262728" watchObservedRunningTime="2026-07-10 22:43:43.891231012 +0000 UTC m=+42.768922633" Jul 10 22:43:43.891567 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.891529 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t4hfp" podStartSLOduration=13.445153905 podStartE2EDuration="42.891523003s" totalImagesPullingTime="29.446369098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:43.889962326 +0000 UTC m=+42.767653975" watchObservedRunningTime="2026-07-10 22:43:43.891523003 +0000 UTC m=+42.769214621" Jul 10 22:43:43.909032 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:43.908992 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gqks5" podStartSLOduration=3.021185877 podStartE2EDuration="8.908961426s" totalImagesPullingTime="5.887775549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:43.908347056 +0000 UTC m=+42.786038675" watchObservedRunningTime="2026-07-10 22:43:43.908961426 +0000 UTC m=+42.786653045" Jul 10 22:43:44.861035 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:44.860930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spnht" event={"ID":"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9","Type":"ContainerStarted","Data":"ba117ec4984e27095a01a27e02b5a21b36ae2aa4d46a51033401af39415af47e"} Jul 10 22:43:46.958875 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:46.958841 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tjqrv"] Jul 10 22:43:47.055579 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.055546 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tjqrv"] Jul 10 22:43:47.055750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.055683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.058836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.058811 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 22:43:47.059814 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.059794 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 22:43:47.059814 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.059800 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 22:43:47.060005 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.059823 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 10 22:43:47.060005 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.059791 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 10 22:43:47.060369 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.060343 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lkzwg\"" Jul 10 22:43:47.224278 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.224195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab0629d-68f0-46da-ac96-caf194bdfa71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.224278 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.224258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab0629d-68f0-46da-ac96-caf194bdfa71-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.224448 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.224285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzjms\" (UniqueName: \"kubernetes.io/projected/dab0629d-68f0-46da-ac96-caf194bdfa71-kube-api-access-pzjms\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.224448 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.224384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dab0629d-68f0-46da-ac96-caf194bdfa71-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.325476 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.325441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab0629d-68f0-46da-ac96-caf194bdfa71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.325476 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.325479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab0629d-68f0-46da-ac96-caf194bdfa71-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.325687 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.325508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzjms\" (UniqueName: \"kubernetes.io/projected/dab0629d-68f0-46da-ac96-caf194bdfa71-kube-api-access-pzjms\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.325687 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.325593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dab0629d-68f0-46da-ac96-caf194bdfa71-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.326145 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.326125 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab0629d-68f0-46da-ac96-caf194bdfa71-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.329210 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.329188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab0629d-68f0-46da-ac96-caf194bdfa71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.329327 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.329277 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dab0629d-68f0-46da-ac96-caf194bdfa71-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.333242 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.333215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzjms\" (UniqueName: \"kubernetes.io/projected/dab0629d-68f0-46da-ac96-caf194bdfa71-kube-api-access-pzjms\") pod \"prometheus-operator-64785b6675-tjqrv\" (UID: \"dab0629d-68f0-46da-ac96-caf194bdfa71\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.366125 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.366101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" Jul 10 22:43:47.508047 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.507955 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tjqrv"] Jul 10 22:43:47.638439 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:47.638396 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab0629d_68f0_46da_ac96_caf194bdfa71.slice/crio-152c3c5236abe2bab2a32b5b758a3c9a366b1f837e096882a7114ab852297f38 WatchSource:0}: Error finding container 152c3c5236abe2bab2a32b5b758a3c9a366b1f837e096882a7114ab852297f38: Status 404 returned error can't find the container with id 152c3c5236abe2bab2a32b5b758a3c9a366b1f837e096882a7114ab852297f38 Jul 10 22:43:47.875389 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.875344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-spnht" event={"ID":"f1988cf5-429e-4b86-8a35-0ee1ddabd8e9","Type":"ContainerStarted","Data":"9f349798e7808b1829d65f1950fa14a8885a23dbdfc1fef1e4ff90fb1929f6ca"} Jul 10 22:43:47.876539 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.876511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" event={"ID":"dab0629d-68f0-46da-ac96-caf194bdfa71","Type":"ContainerStarted","Data":"152c3c5236abe2bab2a32b5b758a3c9a366b1f837e096882a7114ab852297f38"} Jul 10 22:43:47.893111 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:47.893068 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-spnht" podStartSLOduration=2.492743985 podStartE2EDuration="12.893052769s" totalImagesPullingTime="10.400308784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:47.8925455 +0000 UTC m=+46.770237146" watchObservedRunningTime="2026-07-10 22:43:47.893052769 +0000 UTC m=+46.770744388" Jul 10 22:43:49.883659 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:49.883617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" event={"ID":"dab0629d-68f0-46da-ac96-caf194bdfa71","Type":"ContainerStarted","Data":"44204871f34d0b9ea225c1dcaf028b030f376fc016f56f4a67a5ded04a543ab0"} Jul 10 22:43:49.884068 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:49.883685 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" event={"ID":"dab0629d-68f0-46da-ac96-caf194bdfa71","Type":"ContainerStarted","Data":"b1380b5ee2c1754656123df49376e6fd13218aaaa30553721c0d9e92caeb62ce"} Jul 10 22:43:49.902485 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:49.902433 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-tjqrv" podStartSLOduration=2.243098222 podStartE2EDuration="3.902417018s" totalImagesPullingTime="1.659318796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:49.901847877 +0000 UTC m=+48.779539497" watchObservedRunningTime="2026-07-10 22:43:49.902417018 +0000 UTC m=+48.780108638" Jul 10 22:43:51.285275 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.285235 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs"] Jul 10 22:43:51.301451 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.301421 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.302917 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.302885 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs"] Jul 10 22:43:51.304659 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.304640 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 22:43:51.304942 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.304915 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-58ncf\"" Jul 10 22:43:51.304942 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.304921 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 22:43:51.313387 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.313365 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-khttt"] Jul 10 22:43:51.327441 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.327420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.330877 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.330857 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 22:43:51.331164 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.331143 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 22:43:51.331379 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.331362 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 22:43:51.331532 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.331511 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jwtnd\"" Jul 10 22:43:51.456015 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.455958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.456015 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-root\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456264 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-accelerators-collector-config\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456264 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456264 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.456264 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.456264 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bbca213-0413-469a-8f42-3ba9c441b435-metrics-client-ca\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwhd\" (UniqueName: \"kubernetes.io/projected/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-kube-api-access-8bwhd\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.456469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-sys\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-textfile\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjrw\" (UniqueName: \"kubernetes.io/projected/5bbca213-0413-469a-8f42-3ba9c441b435-kube-api-access-vqjrw\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-tls\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.456610 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.456465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-wtmp\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.557654 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.557654 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-root\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.557654 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-accelerators-collector-config\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.557654 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-root\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:51.557724 2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:51.557801 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-tls podName:94ac4c2d-76ab-4d27-8b51-601a33fc8bce nodeName:}" failed. No retries permitted until 2026-07-10 22:43:52.057784517 +0000 UTC m=+50.935476114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-p4zhs" (UID: "94ac4c2d-76ab-4d27-8b51-601a33fc8bce") : secret "openshift-state-metrics-tls" not found Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bbca213-0413-469a-8f42-3ba9c441b435-metrics-client-ca\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwhd\" (UniqueName: \"kubernetes.io/projected/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-kube-api-access-8bwhd\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.557987 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.557987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-sys\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-textfile\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjrw\" (UniqueName: \"kubernetes.io/projected/5bbca213-0413-469a-8f42-3ba9c441b435-kube-api-access-vqjrw\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-tls\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-wtmp\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-accelerators-collector-config\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-wtmp\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bbca213-0413-469a-8f42-3ba9c441b435-sys\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558664 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558641 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-textfile\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.558887 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.558865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bbca213-0413-469a-8f42-3ba9c441b435-metrics-client-ca\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.559082 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.559059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.560422 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.560403 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-tls\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.560528 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.560510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bbca213-0413-469a-8f42-3ba9c441b435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.560683 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.560664 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.566574 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.566549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjrw\" (UniqueName: \"kubernetes.io/projected/5bbca213-0413-469a-8f42-3ba9c441b435-kube-api-access-vqjrw\") pod \"node-exporter-khttt\" (UID: \"5bbca213-0413-469a-8f42-3ba9c441b435\") " pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.567226 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.567204 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwhd\" (UniqueName: \"kubernetes.io/projected/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-kube-api-access-8bwhd\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:51.637150 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.637126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-khttt" Jul 10 22:43:51.673014 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:51.672983 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbca213_0413_469a_8f42_3ba9c441b435.slice/crio-45e09e01d3f903a0b4ba6b55992773ab110716493075d8a43e77fa3c04b2306b WatchSource:0}: Error finding container 45e09e01d3f903a0b4ba6b55992773ab110716493075d8a43e77fa3c04b2306b: Status 404 returned error can't find the container with id 45e09e01d3f903a0b4ba6b55992773ab110716493075d8a43e77fa3c04b2306b Jul 10 22:43:51.891007 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:51.890911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khttt" event={"ID":"5bbca213-0413-469a-8f42-3ba9c441b435","Type":"ContainerStarted","Data":"45e09e01d3f903a0b4ba6b55992773ab110716493075d8a43e77fa3c04b2306b"} Jul 10 22:43:52.062389 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:52.062355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:52.064645 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:52.064626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94ac4c2d-76ab-4d27-8b51-601a33fc8bce-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-p4zhs\" (UID: \"94ac4c2d-76ab-4d27-8b51-601a33fc8bce\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:52.211839 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:52.211799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" Jul 10 22:43:52.369105 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:52.369084 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs"] Jul 10 22:43:52.371654 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:52.371621 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ac4c2d_76ab_4d27_8b51_601a33fc8bce.slice/crio-249ceed43484f20f20a7f0880f8eab036fdd3ddfbf7d516cb59a968bc5491dcd WatchSource:0}: Error finding container 249ceed43484f20f20a7f0880f8eab036fdd3ddfbf7d516cb59a968bc5491dcd: Status 404 returned error can't find the container with id 249ceed43484f20f20a7f0880f8eab036fdd3ddfbf7d516cb59a968bc5491dcd Jul 10 22:43:52.895998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:52.895948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" event={"ID":"94ac4c2d-76ab-4d27-8b51-601a33fc8bce","Type":"ContainerStarted","Data":"2b11f78fe30d641d0b885a82da964f224bbb6e5219f1a98a3faa206ef3f6968f"} Jul 10 22:43:52.896167 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:52.896030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" event={"ID":"94ac4c2d-76ab-4d27-8b51-601a33fc8bce","Type":"ContainerStarted","Data":"3c056e553d5059ab91ce34bc76bc68912dfabf106a60a51d310c5e16a84cc8cc"} Jul 10 22:43:52.896167 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:52.896056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" event={"ID":"94ac4c2d-76ab-4d27-8b51-601a33fc8bce","Type":"ContainerStarted","Data":"249ceed43484f20f20a7f0880f8eab036fdd3ddfbf7d516cb59a968bc5491dcd"} Jul 10 22:43:53.399070 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.399004 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-64c7d58c74-p4s2w"] Jul 10 22:43:53.404867 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.404849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.407709 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.407686 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 22:43:53.407834 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.407706 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 22:43:53.407834 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.407718 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mhkgj\"" Jul 10 22:43:53.407926 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.407867 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 22:43:53.408035 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.408020 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4es1cguh5f8cc\"" Jul 10 22:43:53.408717 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.408703 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 22:43:53.408808 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.408727 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 22:43:53.416454 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.416436 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64c7d58c74-p4s2w"] Jul 10 22:43:53.474142 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-tls\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.474142 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-metrics-client-ca\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.474375 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-grpc-tls\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.474375 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.474375 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4drc\" (UniqueName: \"kubernetes.io/projected/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-kube-api-access-q4drc\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.474375 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.474375 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.474375 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.474369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575328 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-tls\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575477 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-metrics-client-ca\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575552 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-grpc-tls\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575612 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575667 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4drc\" (UniqueName: \"kubernetes.io/projected/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-kube-api-access-q4drc\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575716 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575766 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.575821 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.575772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.576165 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.576142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-metrics-client-ca\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.578497 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.578474 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.579051 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.579006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-grpc-tls\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.579051 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.579031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.579215 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.579162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.579523 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.579498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.579711 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.579688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-secret-thanos-querier-tls\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.584233 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.584211 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4drc\" (UniqueName: \"kubernetes.io/projected/163c693a-a7a5-4dbc-b8d1-352fca68a8c0-kube-api-access-q4drc\") pod \"thanos-querier-64c7d58c74-p4s2w\" (UID: \"163c693a-a7a5-4dbc-b8d1-352fca68a8c0\") " pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.713434 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.713362 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:53.863079 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.863053 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gqks5" Jul 10 22:43:53.885179 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.885143 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64c7d58c74-p4s2w"] Jul 10 22:43:53.901095 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.901059 2577 generic.go:356] "Generic (PLEG): container finished" podID="5bbca213-0413-469a-8f42-3ba9c441b435" containerID="8c5da2b8c52c17b77c0caa4a10b37f7729ba8622158771726b01c455a6464ac8" exitCode=0 Jul 10 22:43:53.901245 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:53.901114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khttt" event={"ID":"5bbca213-0413-469a-8f42-3ba9c441b435","Type":"ContainerDied","Data":"8c5da2b8c52c17b77c0caa4a10b37f7729ba8622158771726b01c455a6464ac8"} Jul 10 22:43:54.037129 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:54.037044 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163c693a_a7a5_4dbc_b8d1_352fca68a8c0.slice/crio-5ae4efb3b4a7e0dbeff73efbb249a1a98acf403004e81678452db16c5420874f WatchSource:0}: Error finding container 5ae4efb3b4a7e0dbeff73efbb249a1a98acf403004e81678452db16c5420874f: Status 404 returned error can't find the container with id 5ae4efb3b4a7e0dbeff73efbb249a1a98acf403004e81678452db16c5420874f Jul 10 22:43:54.907178 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:54.907137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khttt" event={"ID":"5bbca213-0413-469a-8f42-3ba9c441b435","Type":"ContainerStarted","Data":"3ed833fb33af8881665a318237a3a32ed5ddfa5ed52cb70933e236a3479fcb51"} Jul 10 22:43:54.907709 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:54.907206 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khttt" event={"ID":"5bbca213-0413-469a-8f42-3ba9c441b435","Type":"ContainerStarted","Data":"96d46c15a2121472d8f9add91c18d4ac224c6b8e6b525815508ec1b5794e5cd8"} Jul 10 22:43:54.908478 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:54.908455 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" event={"ID":"163c693a-a7a5-4dbc-b8d1-352fca68a8c0","Type":"ContainerStarted","Data":"5ae4efb3b4a7e0dbeff73efbb249a1a98acf403004e81678452db16c5420874f"} Jul 10 22:43:54.910712 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:54.910685 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" event={"ID":"94ac4c2d-76ab-4d27-8b51-601a33fc8bce","Type":"ContainerStarted","Data":"05cada94fc27c87af373c7c94c01303da700860b163219bfa5981320e24fadbf"} Jul 10 22:43:54.931037 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:54.930994 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-khttt" podStartSLOduration=2.413308109 podStartE2EDuration="3.930961616s" totalImagesPullingTime="1.410886674s" totalInitContainerRuntime="106.766833ms" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:54.928719917 +0000 UTC m=+53.806411555" watchObservedRunningTime="2026-07-10 22:43:54.930961616 +0000 UTC m=+53.808653235" Jul 10 22:43:54.945189 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:54.945149 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-p4zhs" podStartSLOduration=2.720370994 podStartE2EDuration="3.945140386s" totalImagesPullingTime="1.224769392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:54.944806656 +0000 UTC m=+53.822498275" watchObservedRunningTime="2026-07-10 22:43:54.945140386 +0000 UTC m=+53.822832005" Jul 10 22:43:55.645866 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.645830 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-87d56d7b-rmrbs"] Jul 10 22:43:55.648204 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.648173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.651024 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.650998 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 22:43:55.651024 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.651017 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 22:43:55.652764 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.652512 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 22:43:55.652764 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.652573 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ntvrb\"" Jul 10 22:43:55.652764 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.652618 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-rskouenvuooa\"" Jul 10 22:43:55.652955 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.652790 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 22:43:55.657412 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.657348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-87d56d7b-rmrbs"] Jul 10 22:43:55.694259 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.694224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-secret-metrics-server-tls\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.694443 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.694286 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9d711ea-268c-4d59-bace-1e3e07e8351b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.694443 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.694324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9d711ea-268c-4d59-bace-1e3e07e8351b-audit-log\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.694443 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.694356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcg9\" (UniqueName: \"kubernetes.io/projected/f9d711ea-268c-4d59-bace-1e3e07e8351b-kube-api-access-phcg9\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.694443 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.694423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-client-ca-bundle\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.694661 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.694527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-secret-metrics-server-client-certs\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.694661 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.694562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9d711ea-268c-4d59-bace-1e3e07e8351b-metrics-server-audit-profiles\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795109 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-secret-metrics-server-client-certs\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795223 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9d711ea-268c-4d59-bace-1e3e07e8351b-metrics-server-audit-profiles\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795223 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-secret-metrics-server-tls\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795310 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9d711ea-268c-4d59-bace-1e3e07e8351b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795310 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9d711ea-268c-4d59-bace-1e3e07e8351b-audit-log\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795310 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcg9\" (UniqueName: \"kubernetes.io/projected/f9d711ea-268c-4d59-bace-1e3e07e8351b-kube-api-access-phcg9\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795454 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-client-ca-bundle\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.795847 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.795678 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f9d711ea-268c-4d59-bace-1e3e07e8351b-audit-log\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.796048 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.796022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9d711ea-268c-4d59-bace-1e3e07e8351b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.796479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.796448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f9d711ea-268c-4d59-bace-1e3e07e8351b-metrics-server-audit-profiles\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.797741 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.797717 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-secret-metrics-server-tls\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.798018 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.797998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-client-ca-bundle\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.798088 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.798013 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f9d711ea-268c-4d59-bace-1e3e07e8351b-secret-metrics-server-client-certs\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.804160 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.804138 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcg9\" (UniqueName: \"kubernetes.io/projected/f9d711ea-268c-4d59-bace-1e3e07e8351b-kube-api-access-phcg9\") pod \"metrics-server-87d56d7b-rmrbs\" (UID: \"f9d711ea-268c-4d59-bace-1e3e07e8351b\") " pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:55.916179 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.916146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" event={"ID":"163c693a-a7a5-4dbc-b8d1-352fca68a8c0","Type":"ContainerStarted","Data":"9a8f49fe11590beecbe94e0e4b11fa735ae1280b642f3fa47d5713d4faf84fe3"} Jul 10 22:43:55.916510 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.916251 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" event={"ID":"163c693a-a7a5-4dbc-b8d1-352fca68a8c0","Type":"ContainerStarted","Data":"b7f1e6130509222fdf989e0ce7418d14476a7f298671fe8bbca8491a778dda19"} Jul 10 22:43:55.916510 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.916281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" event={"ID":"163c693a-a7a5-4dbc-b8d1-352fca68a8c0","Type":"ContainerStarted","Data":"06cc701cda02a8fd7451e1d5f3414a296b5bf36c4d6cd1c39fd8596f511e7bbf"} Jul 10 22:43:55.960261 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:55.960221 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:43:56.077071 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.077038 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd"] Jul 10 22:43:56.079525 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.079506 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.082097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.082076 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cl9bf\"" Jul 10 22:43:56.082187 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.082122 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 22:43:56.092839 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.092812 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd"] Jul 10 22:43:56.094047 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.094030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-87d56d7b-rmrbs"] Jul 10 22:43:56.096607 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:56.096578 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d711ea_268c_4d59_bace_1e3e07e8351b.slice/crio-b3a979ea8a4f119db7780f0829ca80ded7e349bcd3d5260b859d4d8ad376b6b1 WatchSource:0}: Error finding container b3a979ea8a4f119db7780f0829ca80ded7e349bcd3d5260b859d4d8ad376b6b1: Status 404 returned error can't find the container with id b3a979ea8a4f119db7780f0829ca80ded7e349bcd3d5260b859d4d8ad376b6b1 Jul 10 22:43:56.098160 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.098131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vnd\" (UniqueName: \"kubernetes.io/projected/2291b66c-ba7b-4034-be14-39f408b6d1ef-kube-api-access-x9vnd\") pod \"monitoring-plugin-5f68df7f58-q4kkd\" (UID: \"2291b66c-ba7b-4034-be14-39f408b6d1ef\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.098266 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.098208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2291b66c-ba7b-4034-be14-39f408b6d1ef-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q4kkd\" (UID: \"2291b66c-ba7b-4034-be14-39f408b6d1ef\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.199136 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.199099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2291b66c-ba7b-4034-be14-39f408b6d1ef-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q4kkd\" (UID: \"2291b66c-ba7b-4034-be14-39f408b6d1ef\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.199315 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.199150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vnd\" (UniqueName: \"kubernetes.io/projected/2291b66c-ba7b-4034-be14-39f408b6d1ef-kube-api-access-x9vnd\") pod \"monitoring-plugin-5f68df7f58-q4kkd\" (UID: \"2291b66c-ba7b-4034-be14-39f408b6d1ef\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.199315 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:56.199267 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 10 22:43:56.199410 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:43:56.199335 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2291b66c-ba7b-4034-be14-39f408b6d1ef-monitoring-plugin-cert podName:2291b66c-ba7b-4034-be14-39f408b6d1ef nodeName:}" failed. No retries permitted until 2026-07-10 22:43:56.699318334 +0000 UTC m=+55.577009931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2291b66c-ba7b-4034-be14-39f408b6d1ef-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-q4kkd" (UID: "2291b66c-ba7b-4034-be14-39f408b6d1ef") : secret "monitoring-plugin-cert" not found Jul 10 22:43:56.207722 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.207688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vnd\" (UniqueName: \"kubernetes.io/projected/2291b66c-ba7b-4034-be14-39f408b6d1ef-kube-api-access-x9vnd\") pod \"monitoring-plugin-5f68df7f58-q4kkd\" (UID: \"2291b66c-ba7b-4034-be14-39f408b6d1ef\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.703618 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.703583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2291b66c-ba7b-4034-be14-39f408b6d1ef-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q4kkd\" (UID: \"2291b66c-ba7b-4034-be14-39f408b6d1ef\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.706187 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.706159 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2291b66c-ba7b-4034-be14-39f408b6d1ef-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q4kkd\" (UID: \"2291b66c-ba7b-4034-be14-39f408b6d1ef\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:56.921674 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.921645 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" event={"ID":"f9d711ea-268c-4d59-bace-1e3e07e8351b","Type":"ContainerStarted","Data":"b3a979ea8a4f119db7780f0829ca80ded7e349bcd3d5260b859d4d8ad376b6b1"} Jul 10 22:43:56.989633 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:56.989610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:57.128618 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.128588 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd"] Jul 10 22:43:57.132397 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:43:57.132367 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2291b66c_ba7b_4034_be14_39f408b6d1ef.slice/crio-4f6c7185231095fd2c19fbc6b1d162d4696a4ae451f68ade3f6b4e4861e1a3f4 WatchSource:0}: Error finding container 4f6c7185231095fd2c19fbc6b1d162d4696a4ae451f68ade3f6b4e4861e1a3f4: Status 404 returned error can't find the container with id 4f6c7185231095fd2c19fbc6b1d162d4696a4ae451f68ade3f6b4e4861e1a3f4 Jul 10 22:43:57.928519 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.928389 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" event={"ID":"f9d711ea-268c-4d59-bace-1e3e07e8351b","Type":"ContainerStarted","Data":"5ebed0fbbfdda940e7ccd8e3f9805877a9b308e077b55599da1b5972855dade7"} Jul 10 22:43:57.929903 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.929875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" event={"ID":"2291b66c-ba7b-4034-be14-39f408b6d1ef","Type":"ContainerStarted","Data":"4f6c7185231095fd2c19fbc6b1d162d4696a4ae451f68ade3f6b4e4861e1a3f4"} Jul 10 22:43:57.934542 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.934515 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" event={"ID":"163c693a-a7a5-4dbc-b8d1-352fca68a8c0","Type":"ContainerStarted","Data":"f1984005f26fd59ce08b871d9a7cd829f23bc8847a3a389538dea2697c3e155a"} Jul 10 22:43:57.934684 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.934585 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" event={"ID":"163c693a-a7a5-4dbc-b8d1-352fca68a8c0","Type":"ContainerStarted","Data":"b0bfc65204afb9827ea2e10eefffb30c79f4315cf047827ce2a228e09fd9c40e"} Jul 10 22:43:57.934756 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.934707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:43:57.934756 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.934743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" event={"ID":"163c693a-a7a5-4dbc-b8d1-352fca68a8c0","Type":"ContainerStarted","Data":"5f727e87fec1848b764d975718db9f95cc3b76ad5771d59a666541c5664ecaf7"} Jul 10 22:43:57.946203 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.946156 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" podStartSLOduration=1.371334501 podStartE2EDuration="2.946136301s" totalImagesPullingTime="1.5748018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:57.94553011 +0000 UTC m=+56.823221730" watchObservedRunningTime="2026-07-10 22:43:57.946136301 +0000 UTC m=+56.823827921" Jul 10 22:43:57.971489 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:57.971425 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" podStartSLOduration=2.296830004 podStartE2EDuration="4.971404448s" totalImagesPullingTime="2.674574444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:57.971096312 +0000 UTC m=+56.848787932" watchObservedRunningTime="2026-07-10 22:43:57.971404448 +0000 UTC m=+56.849096068" Jul 10 22:43:58.939136 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:58.939098 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" event={"ID":"2291b66c-ba7b-4034-be14-39f408b6d1ef","Type":"ContainerStarted","Data":"68e36e565d328f02325e8b9c5f382b1a79df30de58491ccaf8d42b886a039db7"} Jul 10 22:43:58.959319 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:58.959268 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" podStartSLOduration=1.5097669040000001 podStartE2EDuration="2.95925155s" totalImagesPullingTime="1.449484646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:43:58.95827141 +0000 UTC m=+57.835963054" watchObservedRunningTime="2026-07-10 22:43:58.95925155 +0000 UTC m=+57.836943174" Jul 10 22:43:59.941991 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:59.941950 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:43:59.946689 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:43:59.946671 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q4kkd" Jul 10 22:44:00.805793 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:00.805762 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgsp" Jul 10 22:44:03.945358 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:03.945333 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64c7d58c74-p4s2w" Jul 10 22:44:05.536921 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.536889 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86dc9945c4-9q2w7"] Jul 10 22:44:05.539651 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.539636 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.542426 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.542403 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 22:44:05.542536 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.542473 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 22:44:05.543643 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.543627 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 22:44:05.543854 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.543833 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 22:44:05.543914 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.543897 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 22:44:05.543914 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.543905 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 22:44:05.544046 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.543917 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 22:44:05.544046 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.543936 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xmfbg\"" Jul 10 22:44:05.552071 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.552045 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dc9945c4-9q2w7"] Jul 10 22:44:05.586671 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.586645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-serving-cert\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.586671 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.586676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-oauth-config\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.586828 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.586790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-console-config\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.586828 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.586819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk6ns\" (UniqueName: \"kubernetes.io/projected/d22dd57f-a141-4bda-a02d-b051781bea82-kube-api-access-gk6ns\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.586894 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.586851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-oauth-serving-cert\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.586926 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.586902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-service-ca\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.688186 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.688151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-serving-cert\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.688186 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.688184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-oauth-config\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.688367 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.688232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-console-config\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.688367 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.688249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk6ns\" (UniqueName: \"kubernetes.io/projected/d22dd57f-a141-4bda-a02d-b051781bea82-kube-api-access-gk6ns\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.688367 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.688276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-oauth-serving-cert\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.688367 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.688312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-service-ca\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.688989 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.688957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-oauth-serving-cert\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.689110 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.689026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-console-config\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.689110 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.689071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-service-ca\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.690748 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.690719 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-oauth-config\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.690841 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.690778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-serving-cert\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.696792 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.696767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk6ns\" (UniqueName: \"kubernetes.io/projected/d22dd57f-a141-4bda-a02d-b051781bea82-kube-api-access-gk6ns\") pod \"console-86dc9945c4-9q2w7\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.848897 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.848818 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:05.977427 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:05.977295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dc9945c4-9q2w7"] Jul 10 22:44:05.980303 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:44:05.980277 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22dd57f_a141_4bda_a02d_b051781bea82.slice/crio-081c21633f207954076c107618adba596dc17f31a86fee867183703ebd4be5a1 WatchSource:0}: Error finding container 081c21633f207954076c107618adba596dc17f31a86fee867183703ebd4be5a1: Status 404 returned error can't find the container with id 081c21633f207954076c107618adba596dc17f31a86fee867183703ebd4be5a1 Jul 10 22:44:06.966822 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:06.966781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc9945c4-9q2w7" event={"ID":"d22dd57f-a141-4bda-a02d-b051781bea82","Type":"ContainerStarted","Data":"081c21633f207954076c107618adba596dc17f31a86fee867183703ebd4be5a1"} Jul 10 22:44:07.303457 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.303363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:44:07.306301 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.306278 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 22:44:07.316579 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.316550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4d8088-b537-4bfe-9801-6d6cc21b0939-original-pull-secret\") pod \"global-pull-secret-syncer-n2bws\" (UID: \"8c4d8088-b537-4bfe-9801-6d6cc21b0939\") " pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:44:07.355275 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.355235 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n2bws" Jul 10 22:44:07.404289 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.404258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:44:07.406935 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.406910 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 22:44:07.417461 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.417407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79360d2-84bf-4ebf-b7bc-e1e818e43cb0-metrics-certs\") pod \"network-metrics-daemon-9hnxs\" (UID: \"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0\") " pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:44:07.504729 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.504697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:44:07.507510 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.507485 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 22:44:07.515641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.515613 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n2bws"] Jul 10 22:44:07.518777 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:44:07.518743 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4d8088_b537_4bfe_9801_6d6cc21b0939.slice/crio-d2be445f0c8ea7cfb77709db5806a5ced41f77f03fb9cf4c891a0625e002a3ef WatchSource:0}: Error finding container d2be445f0c8ea7cfb77709db5806a5ced41f77f03fb9cf4c891a0625e002a3ef: Status 404 returned error can't find the container with id d2be445f0c8ea7cfb77709db5806a5ced41f77f03fb9cf4c891a0625e002a3ef Jul 10 22:44:07.518777 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.518754 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 22:44:07.528847 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.528825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8q4z\" (UniqueName: \"kubernetes.io/projected/fba36b3d-f496-4988-b805-2ae5dd458684-kube-api-access-j8q4z\") pod \"network-check-target-b85vc\" (UID: \"fba36b3d-f496-4988-b805-2ae5dd458684\") " pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:44:07.648351 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.648272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vx2kj\"" Jul 10 22:44:07.656438 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.656397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hnxs" Jul 10 22:44:07.666333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.666309 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wwzq4\"" Jul 10 22:44:07.674124 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.674101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:44:07.860727 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.860695 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9hnxs"] Jul 10 22:44:07.864856 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:44:07.864821 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79360d2_84bf_4ebf_b7bc_e1e818e43cb0.slice/crio-a257b71c707f1b31fd6e45fbde0b507d882e1ef575386b918d3a0542e9830c1c WatchSource:0}: Error finding container a257b71c707f1b31fd6e45fbde0b507d882e1ef575386b918d3a0542e9830c1c: Status 404 returned error can't find the container with id a257b71c707f1b31fd6e45fbde0b507d882e1ef575386b918d3a0542e9830c1c Jul 10 22:44:07.888875 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.888811 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b85vc"] Jul 10 22:44:07.893839 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:44:07.893801 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba36b3d_f496_4988_b805_2ae5dd458684.slice/crio-70cdea3c794d49c1d6ced97a135403076eed5afac410a0088256e6c7d90e6845 WatchSource:0}: Error finding container 70cdea3c794d49c1d6ced97a135403076eed5afac410a0088256e6c7d90e6845: Status 404 returned error can't find the container with id 70cdea3c794d49c1d6ced97a135403076eed5afac410a0088256e6c7d90e6845 Jul 10 22:44:07.972460 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.972417 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n2bws" event={"ID":"8c4d8088-b537-4bfe-9801-6d6cc21b0939","Type":"ContainerStarted","Data":"d2be445f0c8ea7cfb77709db5806a5ced41f77f03fb9cf4c891a0625e002a3ef"} Jul 10 22:44:07.973946 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.973914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b85vc" event={"ID":"fba36b3d-f496-4988-b805-2ae5dd458684","Type":"ContainerStarted","Data":"70cdea3c794d49c1d6ced97a135403076eed5afac410a0088256e6c7d90e6845"} Jul 10 22:44:07.975335 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:07.975309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9hnxs" event={"ID":"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0","Type":"ContainerStarted","Data":"a257b71c707f1b31fd6e45fbde0b507d882e1ef575386b918d3a0542e9830c1c"} Jul 10 22:44:09.987418 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:09.987384 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc9945c4-9q2w7" event={"ID":"d22dd57f-a141-4bda-a02d-b051781bea82","Type":"ContainerStarted","Data":"17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1"} Jul 10 22:44:10.004780 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:10.004720 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86dc9945c4-9q2w7" podStartSLOduration=1.6368061310000002 podStartE2EDuration="5.004700603s" totalImagesPullingTime="3.367894472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:44:10.003967602 +0000 UTC m=+68.881659237" watchObservedRunningTime="2026-07-10 22:44:10.004700603 +0000 UTC m=+68.882392203" Jul 10 22:44:10.994247 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:10.994187 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9hnxs" event={"ID":"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0","Type":"ContainerStarted","Data":"cc2b09fa24d70040cc4112339bc627b75b9c6c1cc4dcec6072ab8fbb128aea9e"} Jul 10 22:44:10.994749 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:10.994260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9hnxs" event={"ID":"c79360d2-84bf-4ebf-b7bc-e1e818e43cb0","Type":"ContainerStarted","Data":"b4bb94276b62625ae56fbb4fc2874aa9c0e00a76f4c5a398568ad2656165ee5f"} Jul 10 22:44:11.015396 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:11.015343 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9hnxs" podStartSLOduration=67.882771962 podStartE2EDuration="1m10.015322606s" totalImagesPullingTime="2.132550644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:44:11.01269341 +0000 UTC m=+69.890385060" watchObservedRunningTime="2026-07-10 22:44:11.015322606 +0000 UTC m=+69.893014232" Jul 10 22:44:13.005187 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:13.005143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b85vc" event={"ID":"fba36b3d-f496-4988-b805-2ae5dd458684","Type":"ContainerStarted","Data":"5e28df865ba58fcce145336ba945fa0c9c54de1119e4b4329958d3fa9da1653e"} Jul 10 22:44:13.005654 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:13.005312 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:44:13.006777 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:13.006747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n2bws" event={"ID":"8c4d8088-b537-4bfe-9801-6d6cc21b0939","Type":"ContainerStarted","Data":"cf68e99e675be69a299e0c4b650a574162e981dd8250655d2141f89869fbe8fe"} Jul 10 22:44:13.023514 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:13.023470 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b85vc" podStartSLOduration=67.058595657 podStartE2EDuration="1m12.023455208s" totalImagesPullingTime="4.964859551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:44:13.021574058 +0000 UTC m=+71.899265705" watchObservedRunningTime="2026-07-10 22:44:13.023455208 +0000 UTC m=+71.901146827" Jul 10 22:44:13.039056 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:13.039006 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n2bws" podStartSLOduration=66.701702991 podStartE2EDuration="1m12.038990294s" totalImagesPullingTime="5.337287303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:44:13.037199455 +0000 UTC m=+71.914891074" watchObservedRunningTime="2026-07-10 22:44:13.038990294 +0000 UTC m=+71.916681904" Jul 10 22:44:14.109744 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.109704 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b749695f8-49ftb"] Jul 10 22:44:14.184283 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.184246 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b749695f8-49ftb"] Jul 10 22:44:14.184459 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.184339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.192886 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.192862 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 22:44:14.264861 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.264825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhz25\" (UniqueName: \"kubernetes.io/projected/ce4bb9e7-d706-4222-b25a-7769d2503d0b-kube-api-access-rhz25\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.265050 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.264867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-trusted-ca-bundle\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.265050 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.264897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-service-ca\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.265050 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.265033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-serving-cert\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.265164 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.265069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-oauth-config\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.265164 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.265116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-oauth-serving-cert\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.265164 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.265148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-config\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.366490 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.366414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-serving-cert\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.366490 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.366447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-oauth-config\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.366490 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.366468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-oauth-serving-cert\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.366750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.366729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-config\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.366819 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.366805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhz25\" (UniqueName: \"kubernetes.io/projected/ce4bb9e7-d706-4222-b25a-7769d2503d0b-kube-api-access-rhz25\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.366873 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.366845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-trusted-ca-bundle\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.366922 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.366881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-service-ca\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.367304 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.367283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-oauth-serving-cert\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.367473 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.367450 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-config\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.367680 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.367650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-service-ca\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.367779 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.367691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-trusted-ca-bundle\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.369109 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.369084 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-serving-cert\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.369561 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.369542 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-oauth-config\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.375332 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.375311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhz25\" (UniqueName: \"kubernetes.io/projected/ce4bb9e7-d706-4222-b25a-7769d2503d0b-kube-api-access-rhz25\") pod \"console-b749695f8-49ftb\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.495746 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.495706 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:14.682687 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:14.682659 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b749695f8-49ftb"] Jul 10 22:44:14.685237 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:44:14.685208 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4bb9e7_d706_4222_b25a_7769d2503d0b.slice/crio-dcfd7db6da03286600c986007d096edcb7d6174e2142db906afaa1dce4534e99 WatchSource:0}: Error finding container dcfd7db6da03286600c986007d096edcb7d6174e2142db906afaa1dce4534e99: Status 404 returned error can't find the container with id dcfd7db6da03286600c986007d096edcb7d6174e2142db906afaa1dce4534e99 Jul 10 22:44:15.015346 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.015311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b749695f8-49ftb" event={"ID":"ce4bb9e7-d706-4222-b25a-7769d2503d0b","Type":"ContainerStarted","Data":"eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa"} Jul 10 22:44:15.015544 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.015357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b749695f8-49ftb" event={"ID":"ce4bb9e7-d706-4222-b25a-7769d2503d0b","Type":"ContainerStarted","Data":"dcfd7db6da03286600c986007d096edcb7d6174e2142db906afaa1dce4534e99"} Jul 10 22:44:15.033539 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.033491 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b749695f8-49ftb" podStartSLOduration=1.033476415 podStartE2EDuration="1.033476415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:44:15.032952589 +0000 UTC m=+73.910644239" watchObservedRunningTime="2026-07-10 22:44:15.033476415 +0000 UTC m=+73.911168034" Jul 10 22:44:15.849639 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.849600 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:15.850055 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.849666 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:15.854295 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.854274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:15.960316 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.960287 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:44:15.960477 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:15.960345 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:44:16.022874 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:16.022844 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:24.496317 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:24.496183 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:24.496317 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:24.496251 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:24.500955 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:24.500931 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:25.056445 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:25.056419 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:44:25.110617 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:25.110588 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dc9945c4-9q2w7"] Jul 10 22:44:35.966219 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:35.966186 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:44:35.970181 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:35.970158 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-87d56d7b-rmrbs" Jul 10 22:44:44.012914 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:44.012884 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b85vc" Jul 10 22:44:45.684194 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.684161 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-86b9b8b755-4v26x"] Jul 10 22:44:45.767550 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.767519 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86b9b8b755-4v26x"] Jul 10 22:44:45.767708 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.767651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.770233 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.770210 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 22:44:45.771017 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.770997 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 22:44:45.771112 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.771020 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 22:44:45.771112 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.771004 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 22:44:45.771221 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.770997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sj6vf\"" Jul 10 22:44:45.771369 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.771354 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 22:44:45.776516 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.776499 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 22:44:45.801778 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.801748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-telemeter-client-tls\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.801778 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.801780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-serving-certs-ca-bundle\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.801998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.801799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tzq\" (UniqueName: \"kubernetes.io/projected/01fd1d17-0881-487a-bf3a-66616fe17363-kube-api-access-b5tzq\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.801998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.801818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.801998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.801833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-metrics-client-ca\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.801998 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.801952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-secret-telemeter-client\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.802126 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.802057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-federate-client-tls\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.802126 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.802110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.902606 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-federate-client-tls\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.902606 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.902868 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-telemeter-client-tls\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.902868 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-serving-certs-ca-bundle\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.902868 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tzq\" (UniqueName: \"kubernetes.io/projected/01fd1d17-0881-487a-bf3a-66616fe17363-kube-api-access-b5tzq\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.902868 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.903092 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-metrics-client-ca\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.903092 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.902994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-secret-telemeter-client\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.903804 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.903452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-serving-certs-ca-bundle\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.903804 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.903542 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.904024 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.903862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01fd1d17-0881-487a-bf3a-66616fe17363-metrics-client-ca\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.905733 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.905705 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-federate-client-tls\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.906492 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.906470 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-telemeter-client-tls\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.906999 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.906960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.907101 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.907067 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/01fd1d17-0881-487a-bf3a-66616fe17363-secret-telemeter-client\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:45.911337 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:45.911318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tzq\" (UniqueName: \"kubernetes.io/projected/01fd1d17-0881-487a-bf3a-66616fe17363-kube-api-access-b5tzq\") pod \"telemeter-client-86b9b8b755-4v26x\" (UID: \"01fd1d17-0881-487a-bf3a-66616fe17363\") " pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:46.078787 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:46.078696 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" Jul 10 22:44:46.247504 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:46.247422 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86b9b8b755-4v26x"] Jul 10 22:44:46.251841 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:44:46.251810 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fd1d17_0881_487a_bf3a_66616fe17363.slice/crio-6c848f0e4369380a0d744cfe00df218eacb04c7dc14f96a4d4f0f2dafd9a2b5e WatchSource:0}: Error finding container 6c848f0e4369380a0d744cfe00df218eacb04c7dc14f96a4d4f0f2dafd9a2b5e: Status 404 returned error can't find the container with id 6c848f0e4369380a0d744cfe00df218eacb04c7dc14f96a4d4f0f2dafd9a2b5e Jul 10 22:44:47.132266 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:47.132233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" event={"ID":"01fd1d17-0881-487a-bf3a-66616fe17363","Type":"ContainerStarted","Data":"6c848f0e4369380a0d744cfe00df218eacb04c7dc14f96a4d4f0f2dafd9a2b5e"} Jul 10 22:44:49.143519 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:49.143470 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" event={"ID":"01fd1d17-0881-487a-bf3a-66616fe17363","Type":"ContainerStarted","Data":"dcb747c9504d290549bbede4e4dd5826e1eac2b08f27601e4fed1393ea55fe83"} Jul 10 22:44:50.130882 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.130800 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86dc9945c4-9q2w7" podUID="d22dd57f-a141-4bda-a02d-b051781bea82" containerName="console" containerID="cri-o://17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1" gracePeriod=15 Jul 10 22:44:50.154374 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.154344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" event={"ID":"01fd1d17-0881-487a-bf3a-66616fe17363","Type":"ContainerStarted","Data":"2c4cd54f84e9062369e5d31ff260d9ae79db59b575c0974c4d3ce438ac881d9a"} Jul 10 22:44:50.154680 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.154402 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" event={"ID":"01fd1d17-0881-487a-bf3a-66616fe17363","Type":"ContainerStarted","Data":"3afc950d7a6c576530a1aa9097c5ae948cc24ff6fef1c83004ab35d3f83ca3b6"} Jul 10 22:44:50.180441 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.180386 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-86b9b8b755-4v26x" podStartSLOduration=2.211708198 podStartE2EDuration="5.180370231s" totalImagesPullingTime="2.968662033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:44:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:44:50.179440597 +0000 UTC m=+109.057132239" watchObservedRunningTime="2026-07-10 22:44:50.180370231 +0000 UTC m=+109.058061850" Jul 10 22:44:50.372984 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.372938 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:50.439135 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439105 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk6ns\" (UniqueName: \"kubernetes.io/projected/d22dd57f-a141-4bda-a02d-b051781bea82-kube-api-access-gk6ns\") pod \"d22dd57f-a141-4bda-a02d-b051781bea82\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " Jul 10 22:44:50.439324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439151 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-console-config\") pod \"d22dd57f-a141-4bda-a02d-b051781bea82\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " Jul 10 22:44:50.439324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439169 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-service-ca\") pod \"d22dd57f-a141-4bda-a02d-b051781bea82\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " Jul 10 22:44:50.439324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439195 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-oauth-config\") pod \"d22dd57f-a141-4bda-a02d-b051781bea82\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " Jul 10 22:44:50.439324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439255 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-serving-cert\") pod \"d22dd57f-a141-4bda-a02d-b051781bea82\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " Jul 10 22:44:50.439324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439282 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-oauth-serving-cert\") pod \"d22dd57f-a141-4bda-a02d-b051781bea82\" (UID: \"d22dd57f-a141-4bda-a02d-b051781bea82\") " Jul 10 22:44:50.439881 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439843 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-service-ca" (OuterVolumeSpecName: "service-ca") pod "d22dd57f-a141-4bda-a02d-b051781bea82" (UID: "d22dd57f-a141-4bda-a02d-b051781bea82"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:44:50.439881 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439862 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-console-config" (OuterVolumeSpecName: "console-config") pod "d22dd57f-a141-4bda-a02d-b051781bea82" (UID: "d22dd57f-a141-4bda-a02d-b051781bea82"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:44:50.440085 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.439894 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d22dd57f-a141-4bda-a02d-b051781bea82" (UID: "d22dd57f-a141-4bda-a02d-b051781bea82"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:44:50.441606 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.441578 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d22dd57f-a141-4bda-a02d-b051781bea82" (UID: "d22dd57f-a141-4bda-a02d-b051781bea82"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:44:50.441606 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.441587 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22dd57f-a141-4bda-a02d-b051781bea82-kube-api-access-gk6ns" (OuterVolumeSpecName: "kube-api-access-gk6ns") pod "d22dd57f-a141-4bda-a02d-b051781bea82" (UID: "d22dd57f-a141-4bda-a02d-b051781bea82"). InnerVolumeSpecName "kube-api-access-gk6ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:44:50.441945 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.441927 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d22dd57f-a141-4bda-a02d-b051781bea82" (UID: "d22dd57f-a141-4bda-a02d-b051781bea82"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:44:50.540463 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.540433 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk6ns\" (UniqueName: \"kubernetes.io/projected/d22dd57f-a141-4bda-a02d-b051781bea82-kube-api-access-gk6ns\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:44:50.540463 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.540460 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-console-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:44:50.540463 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.540473 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:44:50.540689 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.540481 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-oauth-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:44:50.540689 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.540492 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d22dd57f-a141-4bda-a02d-b051781bea82-console-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:44:50.540689 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.540500 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d22dd57f-a141-4bda-a02d-b051781bea82-oauth-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:44:50.755349 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.755273 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7f444fd8-jfngr"] Jul 10 22:44:50.755592 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.755579 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22dd57f-a141-4bda-a02d-b051781bea82" containerName="console" Jul 10 22:44:50.755633 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.755593 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22dd57f-a141-4bda-a02d-b051781bea82" containerName="console" Jul 10 22:44:50.755666 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.755659 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22dd57f-a141-4bda-a02d-b051781bea82" containerName="console" Jul 10 22:44:50.757597 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.757574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.770368 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.770338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7f444fd8-jfngr"] Jul 10 22:44:50.843147 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.843114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-config\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.843147 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.843148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-oauth-serving-cert\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.843371 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.843188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-trusted-ca-bundle\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.843371 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.843244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-serving-cert\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.843371 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.843260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-oauth-config\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.843371 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.843278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-service-ca\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.843371 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.843301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzpp\" (UniqueName: \"kubernetes.io/projected/c6c93448-afa3-43e5-a216-a20bdb5768c3-kube-api-access-vvzpp\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.944606 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.944572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-config\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.944606 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.944616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-oauth-serving-cert\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.944842 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.944660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-trusted-ca-bundle\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.944842 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.944698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-serving-cert\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.944842 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.944713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-oauth-config\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.944842 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.944734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-service-ca\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.944842 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.944765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzpp\" (UniqueName: \"kubernetes.io/projected/c6c93448-afa3-43e5-a216-a20bdb5768c3-kube-api-access-vvzpp\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.945446 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.945418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-config\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.945733 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.945709 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-trusted-ca-bundle\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.945913 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.945894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-oauth-serving-cert\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.946094 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.946072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-service-ca\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.947810 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.947397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-serving-cert\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.947810 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.947774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-oauth-config\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:50.953816 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:50.953766 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzpp\" (UniqueName: \"kubernetes.io/projected/c6c93448-afa3-43e5-a216-a20bdb5768c3-kube-api-access-vvzpp\") pod \"console-7b7f444fd8-jfngr\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:51.068041 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.067944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:44:51.166105 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.161812 2577 generic.go:356] "Generic (PLEG): container finished" podID="d22dd57f-a141-4bda-a02d-b051781bea82" containerID="17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1" exitCode=0 Jul 10 22:44:51.166105 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.162740 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc9945c4-9q2w7" Jul 10 22:44:51.166853 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.166618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc9945c4-9q2w7" event={"ID":"d22dd57f-a141-4bda-a02d-b051781bea82","Type":"ContainerDied","Data":"17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1"} Jul 10 22:44:51.166853 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.166680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc9945c4-9q2w7" event={"ID":"d22dd57f-a141-4bda-a02d-b051781bea82","Type":"ContainerDied","Data":"081c21633f207954076c107618adba596dc17f31a86fee867183703ebd4be5a1"} Jul 10 22:44:51.166853 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.166710 2577 scope.go:117] "RemoveContainer" containerID="17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1" Jul 10 22:44:51.178131 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.178072 2577 scope.go:117] "RemoveContainer" containerID="17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1" Jul 10 22:44:51.178477 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:44:51.178431 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1\": container with ID starting with 17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1 not found: ID does not exist" containerID="17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1" Jul 10 22:44:51.178555 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.178489 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1"} err="failed to get container status \"17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1\": rpc error: code = NotFound desc = could not find container \"17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1\": container with ID starting with 17ac953aaac96c5989a5696f346ed41e07abc881752039d92feb4b2e7bc106f1 not found: ID does not exist" Jul 10 22:44:51.187676 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.187602 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dc9945c4-9q2w7"] Jul 10 22:44:51.194100 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.193738 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86dc9945c4-9q2w7"] Jul 10 22:44:51.254641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.254611 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7f444fd8-jfngr"] Jul 10 22:44:51.260329 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:44:51.260297 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c93448_afa3_43e5_a216_a20bdb5768c3.slice/crio-3535aa73e3ae80f96d9d6d7cd5236882168d2bd4160752125b1b21e9aae3c703 WatchSource:0}: Error finding container 3535aa73e3ae80f96d9d6d7cd5236882168d2bd4160752125b1b21e9aae3c703: Status 404 returned error can't find the container with id 3535aa73e3ae80f96d9d6d7cd5236882168d2bd4160752125b1b21e9aae3c703 Jul 10 22:44:51.636105 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:51.636075 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22dd57f-a141-4bda-a02d-b051781bea82" path="/var/lib/kubelet/pods/d22dd57f-a141-4bda-a02d-b051781bea82/volumes" Jul 10 22:44:52.168160 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:52.168117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7f444fd8-jfngr" event={"ID":"c6c93448-afa3-43e5-a216-a20bdb5768c3","Type":"ContainerStarted","Data":"1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d"} Jul 10 22:44:52.168567 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:52.168180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7f444fd8-jfngr" event={"ID":"c6c93448-afa3-43e5-a216-a20bdb5768c3","Type":"ContainerStarted","Data":"3535aa73e3ae80f96d9d6d7cd5236882168d2bd4160752125b1b21e9aae3c703"} Jul 10 22:44:52.188711 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:44:52.188656 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b7f444fd8-jfngr" podStartSLOduration=2.188639694 podStartE2EDuration="2.188639694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:44:52.187175932 +0000 UTC m=+111.064867550" watchObservedRunningTime="2026-07-10 22:44:52.188639694 +0000 UTC m=+111.066331313" Jul 10 22:45:01.068266 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:01.068221 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:45:01.068779 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:01.068297 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:45:01.072856 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:01.072836 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:45:01.203741 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:01.203708 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:45:01.251054 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:01.251023 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b749695f8-49ftb"] Jul 10 22:45:26.271433 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.271372 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b749695f8-49ftb" podUID="ce4bb9e7-d706-4222-b25a-7769d2503d0b" containerName="console" containerID="cri-o://eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa" gracePeriod=15 Jul 10 22:45:26.519663 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.519636 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:45:26.637769 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.637681 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-trusted-ca-bundle\") pod \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " Jul 10 22:45:26.637769 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.637721 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-service-ca\") pod \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " Jul 10 22:45:26.637769 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.637743 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-oauth-config\") pod \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " Jul 10 22:45:26.637769 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.637763 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-oauth-serving-cert\") pod \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " Jul 10 22:45:26.638155 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.637781 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhz25\" (UniqueName: \"kubernetes.io/projected/ce4bb9e7-d706-4222-b25a-7769d2503d0b-kube-api-access-rhz25\") pod \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " Jul 10 22:45:26.638155 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.637801 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-config\") pod \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " Jul 10 22:45:26.638155 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.637861 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-serving-cert\") pod \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\" (UID: \"ce4bb9e7-d706-4222-b25a-7769d2503d0b\") " Jul 10 22:45:26.638335 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.638304 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ce4bb9e7-d706-4222-b25a-7769d2503d0b" (UID: "ce4bb9e7-d706-4222-b25a-7769d2503d0b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:45:26.638386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.638330 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce4bb9e7-d706-4222-b25a-7769d2503d0b" (UID: "ce4bb9e7-d706-4222-b25a-7769d2503d0b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:45:26.638386 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.638308 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce4bb9e7-d706-4222-b25a-7769d2503d0b" (UID: "ce4bb9e7-d706-4222-b25a-7769d2503d0b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:45:26.638469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.638443 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-config" (OuterVolumeSpecName: "console-config") pod "ce4bb9e7-d706-4222-b25a-7769d2503d0b" (UID: "ce4bb9e7-d706-4222-b25a-7769d2503d0b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:45:26.639931 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.639911 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce4bb9e7-d706-4222-b25a-7769d2503d0b" (UID: "ce4bb9e7-d706-4222-b25a-7769d2503d0b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:45:26.640019 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.639955 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce4bb9e7-d706-4222-b25a-7769d2503d0b" (UID: "ce4bb9e7-d706-4222-b25a-7769d2503d0b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:45:26.640087 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.640060 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4bb9e7-d706-4222-b25a-7769d2503d0b-kube-api-access-rhz25" (OuterVolumeSpecName: "kube-api-access-rhz25") pod "ce4bb9e7-d706-4222-b25a-7769d2503d0b" (UID: "ce4bb9e7-d706-4222-b25a-7769d2503d0b"). InnerVolumeSpecName "kube-api-access-rhz25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:45:26.739242 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.739192 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:45:26.739242 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.739233 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-oauth-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:45:26.739461 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.739318 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-oauth-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:45:26.739523 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.739452 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhz25\" (UniqueName: \"kubernetes.io/projected/ce4bb9e7-d706-4222-b25a-7769d2503d0b-kube-api-access-rhz25\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:45:26.739523 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.739480 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:45:26.739523 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.739496 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce4bb9e7-d706-4222-b25a-7769d2503d0b-console-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:45:26.739523 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:26.739510 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4bb9e7-d706-4222-b25a-7769d2503d0b-trusted-ca-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:45:27.293407 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.293370 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce4bb9e7-d706-4222-b25a-7769d2503d0b" containerID="eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa" exitCode=0 Jul 10 22:45:27.293834 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.293415 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b749695f8-49ftb" Jul 10 22:45:27.293834 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.293421 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b749695f8-49ftb" event={"ID":"ce4bb9e7-d706-4222-b25a-7769d2503d0b","Type":"ContainerDied","Data":"eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa"} Jul 10 22:45:27.293834 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.293461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b749695f8-49ftb" event={"ID":"ce4bb9e7-d706-4222-b25a-7769d2503d0b","Type":"ContainerDied","Data":"dcfd7db6da03286600c986007d096edcb7d6174e2142db906afaa1dce4534e99"} Jul 10 22:45:27.293834 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.293480 2577 scope.go:117] "RemoveContainer" containerID="eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa" Jul 10 22:45:27.302218 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.302201 2577 scope.go:117] "RemoveContainer" containerID="eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa" Jul 10 22:45:27.302475 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:45:27.302458 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa\": container with ID starting with eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa not found: ID does not exist" containerID="eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa" Jul 10 22:45:27.302515 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.302487 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa"} err="failed to get container status \"eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa\": rpc error: code = NotFound desc = could not find container \"eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa\": container with ID starting with eeda539ef79d756921484003edcaebb7970a014e0d62a4e6ed09ad83baef03fa not found: ID does not exist" Jul 10 22:45:27.313240 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.313214 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b749695f8-49ftb"] Jul 10 22:45:27.318293 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.318275 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b749695f8-49ftb"] Jul 10 22:45:27.634898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:45:27.634820 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce4bb9e7-d706-4222-b25a-7769d2503d0b" path="/var/lib/kubelet/pods/ce4bb9e7-d706-4222-b25a-7769d2503d0b/volumes" Jul 10 22:46:05.302548 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.302513 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-876b77c67-k6542"] Jul 10 22:46:05.302944 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.302791 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4bb9e7-d706-4222-b25a-7769d2503d0b" containerName="console" Jul 10 22:46:05.302944 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.302801 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4bb9e7-d706-4222-b25a-7769d2503d0b" containerName="console" Jul 10 22:46:05.302944 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.302868 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce4bb9e7-d706-4222-b25a-7769d2503d0b" containerName="console" Jul 10 22:46:05.306861 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.306845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.318863 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.318839 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-876b77c67-k6542"] Jul 10 22:46:05.350740 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.350711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-console-config\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.350888 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.350748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-serving-cert\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.350888 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.350764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-oauth-config\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.350888 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.350792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqnhp\" (UniqueName: \"kubernetes.io/projected/1000010c-cb01-4992-b71f-d0c09728da92-kube-api-access-vqnhp\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.350888 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.350847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-service-ca\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.350888 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.350868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-trusted-ca-bundle\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.351100 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.350898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-oauth-serving-cert\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.451715 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.451682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-console-config\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.451715 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.451721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-serving-cert\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.451954 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.451736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-oauth-config\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.451954 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.451762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqnhp\" (UniqueName: \"kubernetes.io/projected/1000010c-cb01-4992-b71f-d0c09728da92-kube-api-access-vqnhp\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.451954 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.451788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-service-ca\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.451954 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.451814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-trusted-ca-bundle\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.451954 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.451858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-oauth-serving-cert\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.452563 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.452535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-console-config\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.452668 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.452565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-service-ca\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.452668 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.452622 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-oauth-serving-cert\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.452816 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.452790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-trusted-ca-bundle\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.454295 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.454274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-serving-cert\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.454382 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.454296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-oauth-config\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.459530 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.459510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqnhp\" (UniqueName: \"kubernetes.io/projected/1000010c-cb01-4992-b71f-d0c09728da92-kube-api-access-vqnhp\") pod \"console-876b77c67-k6542\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.616041 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.615937 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:05.744712 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:05.744679 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-876b77c67-k6542"] Jul 10 22:46:05.746923 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:46:05.746889 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1000010c_cb01_4992_b71f_d0c09728da92.slice/crio-abc5d051336112cede5851f0d63dcdbbd8597379e7cbd0839d366049834d6bde WatchSource:0}: Error finding container abc5d051336112cede5851f0d63dcdbbd8597379e7cbd0839d366049834d6bde: Status 404 returned error can't find the container with id abc5d051336112cede5851f0d63dcdbbd8597379e7cbd0839d366049834d6bde Jul 10 22:46:06.435657 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:06.435622 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-876b77c67-k6542" event={"ID":"1000010c-cb01-4992-b71f-d0c09728da92","Type":"ContainerStarted","Data":"2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9"} Jul 10 22:46:06.436043 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:06.435671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-876b77c67-k6542" event={"ID":"1000010c-cb01-4992-b71f-d0c09728da92","Type":"ContainerStarted","Data":"abc5d051336112cede5851f0d63dcdbbd8597379e7cbd0839d366049834d6bde"} Jul 10 22:46:06.453173 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:06.453124 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-876b77c67-k6542" podStartSLOduration=1.453108696 podStartE2EDuration="1.453108696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:46:06.452023099 +0000 UTC m=+185.329714743" watchObservedRunningTime="2026-07-10 22:46:06.453108696 +0000 UTC m=+185.330800315" Jul 10 22:46:15.616387 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:15.616340 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:15.616883 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:15.616434 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:15.621040 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:15.621021 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:16.476487 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:16.476461 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:46:16.520829 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:16.520800 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7f444fd8-jfngr"] Jul 10 22:46:39.491725 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.491689 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb"] Jul 10 22:46:39.495173 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.495156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.497987 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.497951 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hfj7n\"" Jul 10 22:46:39.498098 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.497996 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 22:46:39.499112 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.499091 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 22:46:39.506496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.506479 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb"] Jul 10 22:46:39.636547 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.636513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtns\" (UniqueName: \"kubernetes.io/projected/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-kube-api-access-hhtns\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.636724 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.636576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.636724 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.636615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.737677 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.737634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.737876 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.737723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtns\" (UniqueName: \"kubernetes.io/projected/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-kube-api-access-hhtns\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.737876 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.737779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.738142 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.738118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.738142 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.738135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.746508 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.746460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtns\" (UniqueName: \"kubernetes.io/projected/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-kube-api-access-hhtns\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.804148 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.804112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:39.932904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:39.932878 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb"] Jul 10 22:46:39.935381 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:46:39.935351 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e36bd2_5ef9_4ddf_b7ba_27eb9a208499.slice/crio-6418b5182b34031a6a7822a63ab34993de71927eb136f144c137abf8aed9b1fd WatchSource:0}: Error finding container 6418b5182b34031a6a7822a63ab34993de71927eb136f144c137abf8aed9b1fd: Status 404 returned error can't find the container with id 6418b5182b34031a6a7822a63ab34993de71927eb136f144c137abf8aed9b1fd Jul 10 22:46:40.559391 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:40.559352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" event={"ID":"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499","Type":"ContainerStarted","Data":"6418b5182b34031a6a7822a63ab34993de71927eb136f144c137abf8aed9b1fd"} Jul 10 22:46:41.544428 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.544383 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b7f444fd8-jfngr" podUID="c6c93448-afa3-43e5-a216-a20bdb5768c3" containerName="console" containerID="cri-o://1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d" gracePeriod=15 Jul 10 22:46:41.798610 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.798546 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:46:41.958950 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.958919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-service-ca\") pod \"c6c93448-afa3-43e5-a216-a20bdb5768c3\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " Jul 10 22:46:41.959145 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.958992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-oauth-config\") pod \"c6c93448-afa3-43e5-a216-a20bdb5768c3\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " Jul 10 22:46:41.959145 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-oauth-serving-cert\") pod \"c6c93448-afa3-43e5-a216-a20bdb5768c3\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " Jul 10 22:46:41.959145 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959142 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-trusted-ca-bundle\") pod \"c6c93448-afa3-43e5-a216-a20bdb5768c3\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " Jul 10 22:46:41.959308 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959168 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvzpp\" (UniqueName: \"kubernetes.io/projected/c6c93448-afa3-43e5-a216-a20bdb5768c3-kube-api-access-vvzpp\") pod \"c6c93448-afa3-43e5-a216-a20bdb5768c3\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " Jul 10 22:46:41.959308 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-config\") pod \"c6c93448-afa3-43e5-a216-a20bdb5768c3\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " Jul 10 22:46:41.959308 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959264 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-serving-cert\") pod \"c6c93448-afa3-43e5-a216-a20bdb5768c3\" (UID: \"c6c93448-afa3-43e5-a216-a20bdb5768c3\") " Jul 10 22:46:41.959481 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959444 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-service-ca" (OuterVolumeSpecName: "service-ca") pod "c6c93448-afa3-43e5-a216-a20bdb5768c3" (UID: "c6c93448-afa3-43e5-a216-a20bdb5768c3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:46:41.959601 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959582 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:41.959676 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959582 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c6c93448-afa3-43e5-a216-a20bdb5768c3" (UID: "c6c93448-afa3-43e5-a216-a20bdb5768c3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:46:41.959676 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959613 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c6c93448-afa3-43e5-a216-a20bdb5768c3" (UID: "c6c93448-afa3-43e5-a216-a20bdb5768c3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:46:41.959676 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.959652 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-config" (OuterVolumeSpecName: "console-config") pod "c6c93448-afa3-43e5-a216-a20bdb5768c3" (UID: "c6c93448-afa3-43e5-a216-a20bdb5768c3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:46:41.961622 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.961584 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c93448-afa3-43e5-a216-a20bdb5768c3-kube-api-access-vvzpp" (OuterVolumeSpecName: "kube-api-access-vvzpp") pod "c6c93448-afa3-43e5-a216-a20bdb5768c3" (UID: "c6c93448-afa3-43e5-a216-a20bdb5768c3"). InnerVolumeSpecName "kube-api-access-vvzpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:46:41.961815 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.961785 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c6c93448-afa3-43e5-a216-a20bdb5768c3" (UID: "c6c93448-afa3-43e5-a216-a20bdb5768c3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:46:41.961943 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:41.961845 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c6c93448-afa3-43e5-a216-a20bdb5768c3" (UID: "c6c93448-afa3-43e5-a216-a20bdb5768c3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:46:42.060762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.060684 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:42.060762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.060712 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:42.060762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.060720 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6c93448-afa3-43e5-a216-a20bdb5768c3-console-oauth-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:42.060762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.060729 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-oauth-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:42.060762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.060738 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c93448-afa3-43e5-a216-a20bdb5768c3-trusted-ca-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:42.060762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.060746 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzpp\" (UniqueName: \"kubernetes.io/projected/c6c93448-afa3-43e5-a216-a20bdb5768c3-kube-api-access-vvzpp\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:42.569920 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.569882 2577 generic.go:356] "Generic (PLEG): container finished" podID="c6c93448-afa3-43e5-a216-a20bdb5768c3" containerID="1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d" exitCode=0 Jul 10 22:46:42.570116 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.569931 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7f444fd8-jfngr" Jul 10 22:46:42.570116 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.569955 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7f444fd8-jfngr" event={"ID":"c6c93448-afa3-43e5-a216-a20bdb5768c3","Type":"ContainerDied","Data":"1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d"} Jul 10 22:46:42.570116 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.570027 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7f444fd8-jfngr" event={"ID":"c6c93448-afa3-43e5-a216-a20bdb5768c3","Type":"ContainerDied","Data":"3535aa73e3ae80f96d9d6d7cd5236882168d2bd4160752125b1b21e9aae3c703"} Jul 10 22:46:42.570116 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.570058 2577 scope.go:117] "RemoveContainer" containerID="1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d" Jul 10 22:46:42.580073 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.580050 2577 scope.go:117] "RemoveContainer" containerID="1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d" Jul 10 22:46:42.580395 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:46:42.580344 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d\": container with ID starting with 1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d not found: ID does not exist" containerID="1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d" Jul 10 22:46:42.580488 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.580385 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d"} err="failed to get container status \"1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d\": rpc error: code = NotFound desc = could not find container \"1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d\": container with ID starting with 1ee32767d48219c3492b47a4ab8dfdebd7820afcca0fbf879aa34b50e52f349d not found: ID does not exist" Jul 10 22:46:42.594192 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.594167 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7f444fd8-jfngr"] Jul 10 22:46:42.598049 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:42.598026 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b7f444fd8-jfngr"] Jul 10 22:46:43.635395 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:43.635355 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c93448-afa3-43e5-a216-a20bdb5768c3" path="/var/lib/kubelet/pods/c6c93448-afa3-43e5-a216-a20bdb5768c3/volumes" Jul 10 22:46:45.585090 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:45.585001 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerID="4172c6410464142af170f5012c71d5de8f0d9c781bdcd02f710edba205878475" exitCode=0 Jul 10 22:46:45.585090 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:45.585045 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" event={"ID":"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499","Type":"ContainerDied","Data":"4172c6410464142af170f5012c71d5de8f0d9c781bdcd02f710edba205878475"} Jul 10 22:46:48.598716 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:48.598675 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerID="56355be0545597f82cbe21cf7393a91857c1ef415ba66dcd9217f390997c8c1e" exitCode=0 Jul 10 22:46:48.598716 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:48.598716 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" event={"ID":"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499","Type":"ContainerDied","Data":"56355be0545597f82cbe21cf7393a91857c1ef415ba66dcd9217f390997c8c1e"} Jul 10 22:46:58.640984 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:58.640929 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerID="26bcf06fdad1b579dc1ad0203062d68e7bc93cb7ccf6d5664fb0097005742ff8" exitCode=0 Jul 10 22:46:58.641381 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:58.641010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" event={"ID":"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499","Type":"ContainerDied","Data":"26bcf06fdad1b579dc1ad0203062d68e7bc93cb7ccf6d5664fb0097005742ff8"} Jul 10 22:46:59.765100 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.765078 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:46:59.811693 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.811665 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-bundle\") pod \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " Jul 10 22:46:59.811835 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.811757 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhtns\" (UniqueName: \"kubernetes.io/projected/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-kube-api-access-hhtns\") pod \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " Jul 10 22:46:59.811835 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.811796 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-util\") pod \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\" (UID: \"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499\") " Jul 10 22:46:59.812456 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.812429 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-bundle" (OuterVolumeSpecName: "bundle") pod "d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" (UID: "d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:46:59.813868 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.813849 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-kube-api-access-hhtns" (OuterVolumeSpecName: "kube-api-access-hhtns") pod "d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" (UID: "d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499"). InnerVolumeSpecName "kube-api-access-hhtns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:46:59.816435 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.816408 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-util" (OuterVolumeSpecName: "util") pod "d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" (UID: "d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:46:59.913358 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.913331 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:59.913358 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.913359 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhtns\" (UniqueName: \"kubernetes.io/projected/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-kube-api-access-hhtns\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:46:59.913541 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:46:59.913371 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499-util\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:47:00.648786 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:00.648754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" event={"ID":"d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499","Type":"ContainerDied","Data":"6418b5182b34031a6a7822a63ab34993de71927eb136f144c137abf8aed9b1fd"} Jul 10 22:47:00.648786 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:00.648791 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042psb" Jul 10 22:47:00.649002 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:00.648795 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6418b5182b34031a6a7822a63ab34993de71927eb136f144c137abf8aed9b1fd" Jul 10 22:47:07.487029 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.486993 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd"] Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487299 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerName="extract" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487309 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerName="extract" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487324 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerName="util" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487330 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerName="util" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487338 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c93448-afa3-43e5-a216-a20bdb5768c3" containerName="console" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487343 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c93448-afa3-43e5-a216-a20bdb5768c3" containerName="console" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487352 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerName="pull" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487358 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerName="pull" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487403 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e36bd2-5ef9-4ddf-b7ba-27eb9a208499" containerName="extract" Jul 10 22:47:07.487471 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.487414 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c93448-afa3-43e5-a216-a20bdb5768c3" containerName="console" Jul 10 22:47:07.491561 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.491542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.494490 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.494466 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 22:47:07.494625 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.494472 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xgwq4\"" Jul 10 22:47:07.494625 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.494526 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:47:07.502920 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.502898 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd"] Jul 10 22:47:07.576491 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.576452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038b357-ba58-479f-8b85-12bca711b866-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gnkkd\" (UID: \"d038b357-ba58-479f-8b85-12bca711b866\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.576668 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.576521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbktb\" (UniqueName: \"kubernetes.io/projected/d038b357-ba58-479f-8b85-12bca711b866-kube-api-access-vbktb\") pod \"cert-manager-operator-controller-manager-55c444bf86-gnkkd\" (UID: \"d038b357-ba58-479f-8b85-12bca711b866\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.677285 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.677257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbktb\" (UniqueName: \"kubernetes.io/projected/d038b357-ba58-479f-8b85-12bca711b866-kube-api-access-vbktb\") pod \"cert-manager-operator-controller-manager-55c444bf86-gnkkd\" (UID: \"d038b357-ba58-479f-8b85-12bca711b866\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.677473 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.677312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038b357-ba58-479f-8b85-12bca711b866-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gnkkd\" (UID: \"d038b357-ba58-479f-8b85-12bca711b866\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.677667 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.677643 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038b357-ba58-479f-8b85-12bca711b866-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gnkkd\" (UID: \"d038b357-ba58-479f-8b85-12bca711b866\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.686493 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.686462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbktb\" (UniqueName: \"kubernetes.io/projected/d038b357-ba58-479f-8b85-12bca711b866-kube-api-access-vbktb\") pod \"cert-manager-operator-controller-manager-55c444bf86-gnkkd\" (UID: \"d038b357-ba58-479f-8b85-12bca711b866\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.800683 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.800603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" Jul 10 22:47:07.934413 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:07.934266 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd"] Jul 10 22:47:07.937325 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:47:07.937297 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd038b357_ba58_479f_8b85_12bca711b866.slice/crio-f7209b70afb7a053a1c22611933b8c7fdecdf9c67f6320ef90012633cbe28e72 WatchSource:0}: Error finding container f7209b70afb7a053a1c22611933b8c7fdecdf9c67f6320ef90012633cbe28e72: Status 404 returned error can't find the container with id f7209b70afb7a053a1c22611933b8c7fdecdf9c67f6320ef90012633cbe28e72 Jul 10 22:47:08.679146 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:08.679107 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" event={"ID":"d038b357-ba58-479f-8b85-12bca711b866","Type":"ContainerStarted","Data":"f7209b70afb7a053a1c22611933b8c7fdecdf9c67f6320ef90012633cbe28e72"} Jul 10 22:47:09.684453 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:09.684359 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" event={"ID":"d038b357-ba58-479f-8b85-12bca711b866","Type":"ContainerStarted","Data":"1b9ed75d57acbc10345fa06cbe4db5bd64c6d0778eaacb18c15f9c84a9ce858c"} Jul 10 22:47:09.724414 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:09.724367 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gnkkd" podStartSLOduration=1.240606319 podStartE2EDuration="2.724348628s" totalImagesPullingTime="1.483742309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:47:09.721938902 +0000 UTC m=+248.599630518" watchObservedRunningTime="2026-07-10 22:47:09.724348628 +0000 UTC m=+248.602040247" Jul 10 22:47:10.978155 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:10.978116 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q"] Jul 10 22:47:10.980530 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:10.980507 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:10.983269 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:10.983249 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 22:47:10.984538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:10.984511 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hfj7n\"" Jul 10 22:47:10.984654 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:10.984561 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 22:47:10.990160 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:10.990138 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q"] Jul 10 22:47:11.107115 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.107076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.107302 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.107130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5mk\" (UniqueName: \"kubernetes.io/projected/3563190a-4350-4ef5-96a0-877c699cbdec-kube-api-access-vs5mk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.107302 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.107170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.208418 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.208373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.208614 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.208428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5mk\" (UniqueName: \"kubernetes.io/projected/3563190a-4350-4ef5-96a0-877c699cbdec-kube-api-access-vs5mk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.208614 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.208463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.208778 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.208756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.208867 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.208828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.222982 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.222954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5mk\" (UniqueName: \"kubernetes.io/projected/3563190a-4350-4ef5-96a0-877c699cbdec-kube-api-access-vs5mk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.289890 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.289812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:11.421323 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.421294 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q"] Jul 10 22:47:11.422706 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:47:11.422684 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3563190a_4350_4ef5_96a0_877c699cbdec.slice/crio-0738e8fb2d26c3eb68969088b50502f9427a460f873c84615fa7617645d409ff WatchSource:0}: Error finding container 0738e8fb2d26c3eb68969088b50502f9427a460f873c84615fa7617645d409ff: Status 404 returned error can't find the container with id 0738e8fb2d26c3eb68969088b50502f9427a460f873c84615fa7617645d409ff Jul 10 22:47:11.460024 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.459998 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9zd8l"] Jul 10 22:47:11.463531 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.463512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.466817 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.466795 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 22:47:11.466966 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.466819 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 22:47:11.466966 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.466860 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-c84mp\"" Jul 10 22:47:11.476056 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.476037 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9zd8l"] Jul 10 22:47:11.611539 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.611446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9zd8l\" (UID: \"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.611680 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.611553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwj6\" (UniqueName: \"kubernetes.io/projected/2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505-kube-api-access-lzwj6\") pod \"cert-manager-webhook-6486ffd997-9zd8l\" (UID: \"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.692848 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.692810 2577 generic.go:356] "Generic (PLEG): container finished" podID="3563190a-4350-4ef5-96a0-877c699cbdec" containerID="50f02370e31ef85d4b014a6af570d445f7e521e54cd562ca29ab96aeec9a9c1d" exitCode=0 Jul 10 22:47:11.693051 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.692862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" event={"ID":"3563190a-4350-4ef5-96a0-877c699cbdec","Type":"ContainerDied","Data":"50f02370e31ef85d4b014a6af570d445f7e521e54cd562ca29ab96aeec9a9c1d"} Jul 10 22:47:11.693051 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.692911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" event={"ID":"3563190a-4350-4ef5-96a0-877c699cbdec","Type":"ContainerStarted","Data":"0738e8fb2d26c3eb68969088b50502f9427a460f873c84615fa7617645d409ff"} Jul 10 22:47:11.712202 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.712177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwj6\" (UniqueName: \"kubernetes.io/projected/2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505-kube-api-access-lzwj6\") pod \"cert-manager-webhook-6486ffd997-9zd8l\" (UID: \"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.712381 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.712361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9zd8l\" (UID: \"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.724590 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.724564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9zd8l\" (UID: \"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.724751 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.724733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwj6\" (UniqueName: \"kubernetes.io/projected/2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505-kube-api-access-lzwj6\") pod \"cert-manager-webhook-6486ffd997-9zd8l\" (UID: \"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.791672 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.791643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:11.915101 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:11.915067 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9zd8l"] Jul 10 22:47:11.917893 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:47:11.917867 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d4ce2a5_b8b2_47f3_96c4_c655c5a5b505.slice/crio-2a876869242c55fabcad06bb5f82dd06f3955d7c6740e6732722ea2321f21349 WatchSource:0}: Error finding container 2a876869242c55fabcad06bb5f82dd06f3955d7c6740e6732722ea2321f21349: Status 404 returned error can't find the container with id 2a876869242c55fabcad06bb5f82dd06f3955d7c6740e6732722ea2321f21349 Jul 10 22:47:12.700217 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:12.700170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" event={"ID":"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505","Type":"ContainerStarted","Data":"2a876869242c55fabcad06bb5f82dd06f3955d7c6740e6732722ea2321f21349"} Jul 10 22:47:13.436741 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.436709 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nghd5"] Jul 10 22:47:13.439097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.439073 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.441547 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.441527 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lnrns\"" Jul 10 22:47:13.449024 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.449002 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nghd5"] Jul 10 22:47:13.528819 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.528788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4cl\" (UniqueName: \"kubernetes.io/projected/8f9e1ff7-8dbc-450d-884a-ac04879a234f-kube-api-access-jr4cl\") pod \"cert-manager-cainjector-559cdd8c87-nghd5\" (UID: \"8f9e1ff7-8dbc-450d-884a-ac04879a234f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.528927 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.528905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9e1ff7-8dbc-450d-884a-ac04879a234f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nghd5\" (UID: \"8f9e1ff7-8dbc-450d-884a-ac04879a234f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.630167 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.630139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9e1ff7-8dbc-450d-884a-ac04879a234f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nghd5\" (UID: \"8f9e1ff7-8dbc-450d-884a-ac04879a234f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.630298 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.630186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4cl\" (UniqueName: \"kubernetes.io/projected/8f9e1ff7-8dbc-450d-884a-ac04879a234f-kube-api-access-jr4cl\") pod \"cert-manager-cainjector-559cdd8c87-nghd5\" (UID: \"8f9e1ff7-8dbc-450d-884a-ac04879a234f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.645480 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.645453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4cl\" (UniqueName: \"kubernetes.io/projected/8f9e1ff7-8dbc-450d-884a-ac04879a234f-kube-api-access-jr4cl\") pod \"cert-manager-cainjector-559cdd8c87-nghd5\" (UID: \"8f9e1ff7-8dbc-450d-884a-ac04879a234f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.646013 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.645995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9e1ff7-8dbc-450d-884a-ac04879a234f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nghd5\" (UID: \"8f9e1ff7-8dbc-450d-884a-ac04879a234f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.705601 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.705522 2577 generic.go:356] "Generic (PLEG): container finished" podID="3563190a-4350-4ef5-96a0-877c699cbdec" containerID="2bd1abd36ace4cfdc62f31d8f24d188cc3f92d5ca31a9eb6250d0bce86274f18" exitCode=0 Jul 10 22:47:13.705601 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.705597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" event={"ID":"3563190a-4350-4ef5-96a0-877c699cbdec","Type":"ContainerDied","Data":"2bd1abd36ace4cfdc62f31d8f24d188cc3f92d5ca31a9eb6250d0bce86274f18"} Jul 10 22:47:13.748832 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.748809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" Jul 10 22:47:13.904269 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:13.904240 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nghd5"] Jul 10 22:47:13.909454 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:47:13.909434 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f9e1ff7_8dbc_450d_884a_ac04879a234f.slice/crio-607c32e4455fa60aec0f861e15183baa738db116fa3cb97d7665e8d9cd2c4b1a WatchSource:0}: Error finding container 607c32e4455fa60aec0f861e15183baa738db116fa3cb97d7665e8d9cd2c4b1a: Status 404 returned error can't find the container with id 607c32e4455fa60aec0f861e15183baa738db116fa3cb97d7665e8d9cd2c4b1a Jul 10 22:47:14.711778 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:14.711743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" event={"ID":"8f9e1ff7-8dbc-450d-884a-ac04879a234f","Type":"ContainerStarted","Data":"607c32e4455fa60aec0f861e15183baa738db116fa3cb97d7665e8d9cd2c4b1a"} Jul 10 22:47:14.714060 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:14.714030 2577 generic.go:356] "Generic (PLEG): container finished" podID="3563190a-4350-4ef5-96a0-877c699cbdec" containerID="954befe241736df529843a083a1ffebaa1194944aa613d3550e55d04607581fb" exitCode=0 Jul 10 22:47:14.714179 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:14.714071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" event={"ID":"3563190a-4350-4ef5-96a0-877c699cbdec","Type":"ContainerDied","Data":"954befe241736df529843a083a1ffebaa1194944aa613d3550e55d04607581fb"} Jul 10 22:47:16.499019 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.499000 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:16.661842 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.661811 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-util\") pod \"3563190a-4350-4ef5-96a0-877c699cbdec\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " Jul 10 22:47:16.662018 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.661905 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs5mk\" (UniqueName: \"kubernetes.io/projected/3563190a-4350-4ef5-96a0-877c699cbdec-kube-api-access-vs5mk\") pod \"3563190a-4350-4ef5-96a0-877c699cbdec\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " Jul 10 22:47:16.662018 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.661955 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-bundle\") pod \"3563190a-4350-4ef5-96a0-877c699cbdec\" (UID: \"3563190a-4350-4ef5-96a0-877c699cbdec\") " Jul 10 22:47:16.662344 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.662317 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-bundle" (OuterVolumeSpecName: "bundle") pod "3563190a-4350-4ef5-96a0-877c699cbdec" (UID: "3563190a-4350-4ef5-96a0-877c699cbdec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:47:16.664108 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.664079 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3563190a-4350-4ef5-96a0-877c699cbdec-kube-api-access-vs5mk" (OuterVolumeSpecName: "kube-api-access-vs5mk") pod "3563190a-4350-4ef5-96a0-877c699cbdec" (UID: "3563190a-4350-4ef5-96a0-877c699cbdec"). InnerVolumeSpecName "kube-api-access-vs5mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:47:16.692842 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.692803 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-util" (OuterVolumeSpecName: "util") pod "3563190a-4350-4ef5-96a0-877c699cbdec" (UID: "3563190a-4350-4ef5-96a0-877c699cbdec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:47:16.725452 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.725412 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" event={"ID":"8f9e1ff7-8dbc-450d-884a-ac04879a234f","Type":"ContainerStarted","Data":"736d659ac23fc7d02c3bf9bfce387d7576a3a741f893ec69f589d15c518ffe61"} Jul 10 22:47:16.727028 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.726997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" event={"ID":"2d4ce2a5-b8b2-47f3-96c4-c655c5a5b505","Type":"ContainerStarted","Data":"83c56fb287185062b157b1457a0b700422b578c825f5d81cdfa938b522407499"} Jul 10 22:47:16.727177 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.727068 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:16.728906 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.728874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" event={"ID":"3563190a-4350-4ef5-96a0-877c699cbdec","Type":"ContainerDied","Data":"0738e8fb2d26c3eb68969088b50502f9427a460f873c84615fa7617645d409ff"} Jul 10 22:47:16.728906 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.728891 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql78q" Jul 10 22:47:16.729074 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.728922 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0738e8fb2d26c3eb68969088b50502f9427a460f873c84615fa7617645d409ff" Jul 10 22:47:16.761505 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.761454 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nghd5" podStartSLOduration=1.1507608089999999 podStartE2EDuration="3.761437056s" totalImagesPullingTime="2.610676247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:47:16.759055421 +0000 UTC m=+255.636747040" watchObservedRunningTime="2026-07-10 22:47:16.761437056 +0000 UTC m=+255.639128675" Jul 10 22:47:16.763321 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.763300 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-util\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:47:16.763321 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.763322 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs5mk\" (UniqueName: \"kubernetes.io/projected/3563190a-4350-4ef5-96a0-877c699cbdec-kube-api-access-vs5mk\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:47:16.763461 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.763333 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3563190a-4350-4ef5-96a0-877c699cbdec-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:47:16.808868 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:16.808806 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" podStartSLOduration=1.207827334 podStartE2EDuration="5.808783479s" totalImagesPullingTime="4.600956145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:47:16.806250341 +0000 UTC m=+255.683941961" watchObservedRunningTime="2026-07-10 22:47:16.808783479 +0000 UTC m=+255.686475099" Jul 10 22:47:22.211517 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211482 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc"] Jul 10 22:47:22.211904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211780 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3563190a-4350-4ef5-96a0-877c699cbdec" containerName="extract" Jul 10 22:47:22.211904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211791 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3563190a-4350-4ef5-96a0-877c699cbdec" containerName="extract" Jul 10 22:47:22.211904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211801 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3563190a-4350-4ef5-96a0-877c699cbdec" containerName="util" Jul 10 22:47:22.211904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211807 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3563190a-4350-4ef5-96a0-877c699cbdec" containerName="util" Jul 10 22:47:22.211904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211821 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3563190a-4350-4ef5-96a0-877c699cbdec" containerName="pull" Jul 10 22:47:22.211904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211826 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3563190a-4350-4ef5-96a0-877c699cbdec" containerName="pull" Jul 10 22:47:22.211904 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.211879 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3563190a-4350-4ef5-96a0-877c699cbdec" containerName="extract" Jul 10 22:47:22.216610 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.216594 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.219459 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.219437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-x4h8w\"" Jul 10 22:47:22.219841 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.219824 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:47:22.220622 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.220603 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 22:47:22.226320 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.226298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc"] Jul 10 22:47:22.313744 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.313710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2dsgc\" (UID: \"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.313744 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.313748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdr9c\" (UniqueName: \"kubernetes.io/projected/ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826-kube-api-access-gdr9c\") pod \"openshift-lws-operator-fd8ccff4c-2dsgc\" (UID: \"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.414317 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.414278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2dsgc\" (UID: \"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.414485 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.414332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdr9c\" (UniqueName: \"kubernetes.io/projected/ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826-kube-api-access-gdr9c\") pod \"openshift-lws-operator-fd8ccff4c-2dsgc\" (UID: \"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.414754 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.414735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2dsgc\" (UID: \"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.423475 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.423451 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdr9c\" (UniqueName: \"kubernetes.io/projected/ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826-kube-api-access-gdr9c\") pod \"openshift-lws-operator-fd8ccff4c-2dsgc\" (UID: \"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.525794 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.525692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" Jul 10 22:47:22.663272 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.663184 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc"] Jul 10 22:47:22.666058 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:47:22.666033 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab44585a_c7d4_4a1b_81cd_bdd7b0b2c826.slice/crio-56ce05823b793011ac44de3a40237642b6b5e03f9165f5dba787c7f078a9b501 WatchSource:0}: Error finding container 56ce05823b793011ac44de3a40237642b6b5e03f9165f5dba787c7f078a9b501: Status 404 returned error can't find the container with id 56ce05823b793011ac44de3a40237642b6b5e03f9165f5dba787c7f078a9b501 Jul 10 22:47:22.735811 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.735784 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9zd8l" Jul 10 22:47:22.756322 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:22.756290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" event={"ID":"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826","Type":"ContainerStarted","Data":"56ce05823b793011ac44de3a40237642b6b5e03f9165f5dba787c7f078a9b501"} Jul 10 22:47:24.768037 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:24.768003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" event={"ID":"ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826","Type":"ContainerStarted","Data":"88472cf15bad236fbe27f4785119eb16604bf4b1b341e3e5723039f3210faad1"} Jul 10 22:47:24.792326 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:24.792269 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2dsgc" podStartSLOduration=1.146714941 podStartE2EDuration="2.792252339s" totalImagesPullingTime="1.645537398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:47:24.790663881 +0000 UTC m=+263.668355501" watchObservedRunningTime="2026-07-10 22:47:24.792252339 +0000 UTC m=+263.669943959" Jul 10 22:47:27.215102 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.215065 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt"] Jul 10 22:47:27.219603 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.219587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.222129 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.222107 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 22:47:27.222223 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.222110 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hfj7n\"" Jul 10 22:47:27.223116 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.223091 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 22:47:27.226761 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.226462 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt"] Jul 10 22:47:27.358925 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.358896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.359105 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.358958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.359105 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.359032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnfnq\" (UniqueName: \"kubernetes.io/projected/3496d630-9418-45af-a28d-de0c8b04e65c-kube-api-access-dnfnq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.460180 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.460144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.460180 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.460179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfnq\" (UniqueName: \"kubernetes.io/projected/3496d630-9418-45af-a28d-de0c8b04e65c-kube-api-access-dnfnq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.460376 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.460320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.460540 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.460520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.460601 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.460583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.468680 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.468623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfnq\" (UniqueName: \"kubernetes.io/projected/3496d630-9418-45af-a28d-de0c8b04e65c-kube-api-access-dnfnq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.529531 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.529492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:27.663047 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.663023 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt"] Jul 10 22:47:27.665608 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:47:27.665577 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3496d630_9418_45af_a28d_de0c8b04e65c.slice/crio-c3afe483ebfa105a69c7cc814a42dfc2312a0fdc532b8d8a8d010dd6aa9d0393 WatchSource:0}: Error finding container c3afe483ebfa105a69c7cc814a42dfc2312a0fdc532b8d8a8d010dd6aa9d0393: Status 404 returned error can't find the container with id c3afe483ebfa105a69c7cc814a42dfc2312a0fdc532b8d8a8d010dd6aa9d0393 Jul 10 22:47:27.782656 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.782627 2577 generic.go:356] "Generic (PLEG): container finished" podID="3496d630-9418-45af-a28d-de0c8b04e65c" containerID="07283480aadbfbb254b55aa305d82c751a04e04aba95a2c0212930eb85fe5123" exitCode=0 Jul 10 22:47:27.782785 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.782663 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" event={"ID":"3496d630-9418-45af-a28d-de0c8b04e65c","Type":"ContainerDied","Data":"07283480aadbfbb254b55aa305d82c751a04e04aba95a2c0212930eb85fe5123"} Jul 10 22:47:27.782785 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:27.782699 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" event={"ID":"3496d630-9418-45af-a28d-de0c8b04e65c","Type":"ContainerStarted","Data":"c3afe483ebfa105a69c7cc814a42dfc2312a0fdc532b8d8a8d010dd6aa9d0393"} Jul 10 22:47:28.790287 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:28.790243 2577 generic.go:356] "Generic (PLEG): container finished" podID="3496d630-9418-45af-a28d-de0c8b04e65c" containerID="afe42d3941ba929cf2272297ca6b488e09e595761e237d5d9bab0cf49d80b642" exitCode=0 Jul 10 22:47:28.790799 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:28.790407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" event={"ID":"3496d630-9418-45af-a28d-de0c8b04e65c","Type":"ContainerDied","Data":"afe42d3941ba929cf2272297ca6b488e09e595761e237d5d9bab0cf49d80b642"} Jul 10 22:47:29.796527 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:29.796491 2577 generic.go:356] "Generic (PLEG): container finished" podID="3496d630-9418-45af-a28d-de0c8b04e65c" containerID="dbcd1ac56ee7139435010cbad0144bce614a391f4f416555fce360383b744e20" exitCode=0 Jul 10 22:47:29.796897 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:29.796580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" event={"ID":"3496d630-9418-45af-a28d-de0c8b04e65c","Type":"ContainerDied","Data":"dbcd1ac56ee7139435010cbad0144bce614a391f4f416555fce360383b744e20"} Jul 10 22:47:30.961751 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:30.961719 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:30.988562 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:30.988534 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-bundle\") pod \"3496d630-9418-45af-a28d-de0c8b04e65c\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " Jul 10 22:47:30.988730 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:30.988581 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-util\") pod \"3496d630-9418-45af-a28d-de0c8b04e65c\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " Jul 10 22:47:30.988730 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:30.988618 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnfnq\" (UniqueName: \"kubernetes.io/projected/3496d630-9418-45af-a28d-de0c8b04e65c-kube-api-access-dnfnq\") pod \"3496d630-9418-45af-a28d-de0c8b04e65c\" (UID: \"3496d630-9418-45af-a28d-de0c8b04e65c\") " Jul 10 22:47:30.989388 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:30.989331 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-bundle" (OuterVolumeSpecName: "bundle") pod "3496d630-9418-45af-a28d-de0c8b04e65c" (UID: "3496d630-9418-45af-a28d-de0c8b04e65c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:47:30.990957 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:30.990935 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3496d630-9418-45af-a28d-de0c8b04e65c-kube-api-access-dnfnq" (OuterVolumeSpecName: "kube-api-access-dnfnq") pod "3496d630-9418-45af-a28d-de0c8b04e65c" (UID: "3496d630-9418-45af-a28d-de0c8b04e65c"). InnerVolumeSpecName "kube-api-access-dnfnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:47:30.997585 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:30.997561 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-util" (OuterVolumeSpecName: "util") pod "3496d630-9418-45af-a28d-de0c8b04e65c" (UID: "3496d630-9418-45af-a28d-de0c8b04e65c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:47:31.089853 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:31.089773 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:47:31.089853 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:31.089799 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3496d630-9418-45af-a28d-de0c8b04e65c-util\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:47:31.089853 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:31.089808 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnfnq\" (UniqueName: \"kubernetes.io/projected/3496d630-9418-45af-a28d-de0c8b04e65c-kube-api-access-dnfnq\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:47:31.807291 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:31.807258 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" event={"ID":"3496d630-9418-45af-a28d-de0c8b04e65c","Type":"ContainerDied","Data":"c3afe483ebfa105a69c7cc814a42dfc2312a0fdc532b8d8a8d010dd6aa9d0393"} Jul 10 22:47:31.807453 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:31.807306 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3afe483ebfa105a69c7cc814a42dfc2312a0fdc532b8d8a8d010dd6aa9d0393" Jul 10 22:47:31.807453 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:31.807265 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflm6zt" Jul 10 22:47:41.303139 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303106 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq"] Jul 10 22:47:41.303518 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303388 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496d630-9418-45af-a28d-de0c8b04e65c" containerName="util" Jul 10 22:47:41.303518 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303398 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496d630-9418-45af-a28d-de0c8b04e65c" containerName="util" Jul 10 22:47:41.303518 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303406 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496d630-9418-45af-a28d-de0c8b04e65c" containerName="extract" Jul 10 22:47:41.303518 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303413 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496d630-9418-45af-a28d-de0c8b04e65c" containerName="extract" Jul 10 22:47:41.303518 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303431 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496d630-9418-45af-a28d-de0c8b04e65c" containerName="pull" Jul 10 22:47:41.303518 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303437 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496d630-9418-45af-a28d-de0c8b04e65c" containerName="pull" Jul 10 22:47:41.303518 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.303497 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3496d630-9418-45af-a28d-de0c8b04e65c" containerName="extract" Jul 10 22:47:41.305993 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.305952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.309088 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.309061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 22:47:41.309088 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.309081 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 22:47:41.309260 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.309118 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l7nx4\"" Jul 10 22:47:41.309587 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.309567 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 22:47:41.309690 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.309676 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 22:47:41.330597 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.330569 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq"] Jul 10 22:47:41.367807 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.367769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c03c1145-2d83-4a8e-8b3e-9c913005414f-webhook-cert\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.367961 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.367835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c03c1145-2d83-4a8e-8b3e-9c913005414f-manifests\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.367961 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.367880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c03c1145-2d83-4a8e-8b3e-9c913005414f-tmp\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.367961 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.367909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8hp\" (UniqueName: \"kubernetes.io/projected/c03c1145-2d83-4a8e-8b3e-9c913005414f-kube-api-access-vz8hp\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.367961 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.367930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c03c1145-2d83-4a8e-8b3e-9c913005414f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.468320 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.468285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c03c1145-2d83-4a8e-8b3e-9c913005414f-webhook-cert\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.468494 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.468346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c03c1145-2d83-4a8e-8b3e-9c913005414f-manifests\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.468494 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.468378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c03c1145-2d83-4a8e-8b3e-9c913005414f-tmp\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.468494 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.468406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8hp\" (UniqueName: \"kubernetes.io/projected/c03c1145-2d83-4a8e-8b3e-9c913005414f-kube-api-access-vz8hp\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.468494 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.468434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c03c1145-2d83-4a8e-8b3e-9c913005414f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.468750 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.468727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c03c1145-2d83-4a8e-8b3e-9c913005414f-manifests\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.470630 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.470611 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c03c1145-2d83-4a8e-8b3e-9c913005414f-tmp\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.470711 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.470679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c03c1145-2d83-4a8e-8b3e-9c913005414f-webhook-cert\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.470835 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.470818 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c03c1145-2d83-4a8e-8b3e-9c913005414f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.476586 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.476565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8hp\" (UniqueName: \"kubernetes.io/projected/c03c1145-2d83-4a8e-8b3e-9c913005414f-kube-api-access-vz8hp\") pod \"opendatahub-operator-controller-manager-845b6c684-n9pcq\" (UID: \"c03c1145-2d83-4a8e-8b3e-9c913005414f\") " pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.617084 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.617005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:41.756596 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.756569 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq"] Jul 10 22:47:41.759436 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:47:41.759411 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03c1145_2d83_4a8e_8b3e_9c913005414f.slice/crio-89a0097754977dc081608a2fbae26f03aaef87e1ca63eaa8289532a185ae938f WatchSource:0}: Error finding container 89a0097754977dc081608a2fbae26f03aaef87e1ca63eaa8289532a185ae938f: Status 404 returned error can't find the container with id 89a0097754977dc081608a2fbae26f03aaef87e1ca63eaa8289532a185ae938f Jul 10 22:47:41.855075 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:41.855032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" event={"ID":"c03c1145-2d83-4a8e-8b3e-9c913005414f","Type":"ContainerStarted","Data":"89a0097754977dc081608a2fbae26f03aaef87e1ca63eaa8289532a185ae938f"} Jul 10 22:47:44.871612 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:44.871576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" event={"ID":"c03c1145-2d83-4a8e-8b3e-9c913005414f","Type":"ContainerStarted","Data":"4c63581f06b7fb181c1276b0218839d50bdfcf0ee0b1e85077940753c588a4fe"} Jul 10 22:47:45.877239 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:45.877209 2577 generic.go:356] "Generic (PLEG): container finished" podID="c03c1145-2d83-4a8e-8b3e-9c913005414f" containerID="4c63581f06b7fb181c1276b0218839d50bdfcf0ee0b1e85077940753c588a4fe" exitCode=0 Jul 10 22:47:45.877649 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:45.877305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" event={"ID":"c03c1145-2d83-4a8e-8b3e-9c913005414f","Type":"ContainerDied","Data":"4c63581f06b7fb181c1276b0218839d50bdfcf0ee0b1e85077940753c588a4fe"} Jul 10 22:47:46.884580 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:46.884544 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" event={"ID":"c03c1145-2d83-4a8e-8b3e-9c913005414f","Type":"ContainerStarted","Data":"f969b15cb4d7a64d75fe6bbea72c342ad16609ed27814621662a2ff4d6e7761a"} Jul 10 22:47:46.885050 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:46.884691 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:47:46.924675 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:46.924626 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" podStartSLOduration=1.6887312570000002 podStartE2EDuration="5.924610366s" totalImagesPullingTime="3.180914578s" totalInitContainerRuntime="1.054964531s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:47:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:47:46.922851931 +0000 UTC m=+285.800543569" watchObservedRunningTime="2026-07-10 22:47:46.924610366 +0000 UTC m=+285.802301985" Jul 10 22:47:57.891609 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:47:57.891578 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-845b6c684-n9pcq" Jul 10 22:48:00.266862 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.266824 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4"] Jul 10 22:48:00.270780 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.270763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.277168 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.277011 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 22:48:00.277168 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.277010 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4pml6\"" Jul 10 22:48:00.277168 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.277119 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 22:48:00.277168 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.277071 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 22:48:00.287534 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.287509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4"] Jul 10 22:48:00.326733 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.326696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4grt\" (UniqueName: \"kubernetes.io/projected/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-kube-api-access-l4grt\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.326898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.326780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-cert\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.326898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.326810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-metrics-cert\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.326898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.326845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-manager-config\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.427763 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.427726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-cert\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.427763 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.427765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-metrics-cert\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.427966 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.427790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-manager-config\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.427966 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.427868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4grt\" (UniqueName: \"kubernetes.io/projected/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-kube-api-access-l4grt\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.428653 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.428624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-manager-config\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.430309 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.430285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-cert\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.430394 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.430364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-metrics-cert\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.441825 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.441803 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4grt\" (UniqueName: \"kubernetes.io/projected/e5549f83-0ca9-4147-a4f9-e45b5fab53fb-kube-api-access-l4grt\") pod \"lws-controller-manager-7c7f744b8d-xlkc4\" (UID: \"e5549f83-0ca9-4147-a4f9-e45b5fab53fb\") " pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.579757 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.579675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:00.731857 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.731826 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4"] Jul 10 22:48:00.734968 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:48:00.734943 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5549f83_0ca9_4147_a4f9_e45b5fab53fb.slice/crio-cd9aee71c23dbaafd24226ab2c20cee63dd3f5c562cffef2ab4e92b61137755e WatchSource:0}: Error finding container cd9aee71c23dbaafd24226ab2c20cee63dd3f5c562cffef2ab4e92b61137755e: Status 404 returned error can't find the container with id cd9aee71c23dbaafd24226ab2c20cee63dd3f5c562cffef2ab4e92b61137755e Jul 10 22:48:00.950222 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:00.950177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" event={"ID":"e5549f83-0ca9-4147-a4f9-e45b5fab53fb","Type":"ContainerStarted","Data":"cd9aee71c23dbaafd24226ab2c20cee63dd3f5c562cffef2ab4e92b61137755e"} Jul 10 22:48:01.497273 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:01.497244 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:48:01.498362 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:01.498337 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:48:01.503439 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:01.503417 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 22:48:02.961678 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:02.961633 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" event={"ID":"e5549f83-0ca9-4147-a4f9-e45b5fab53fb","Type":"ContainerStarted","Data":"167e2ff2eaf5b0b7e46af3655d066a53325f00a113231e82674795c58aa534ce"} Jul 10 22:48:02.962163 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:02.961792 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:02.981413 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:02.981359 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" podStartSLOduration=1.16861149 podStartE2EDuration="2.981342151s" totalImagesPullingTime="1.812730661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:48:02.979505982 +0000 UTC m=+301.857197634" watchObservedRunningTime="2026-07-10 22:48:02.981342151 +0000 UTC m=+301.859033770" Jul 10 22:48:13.692413 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.692380 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59"] Jul 10 22:48:13.696908 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.696891 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.700677 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.700644 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nwvdk\"" Jul 10 22:48:13.700803 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.700787 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 22:48:13.701188 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.701171 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 22:48:13.737640 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.737609 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 22:48:13.737640 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.737632 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 22:48:13.737829 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.737644 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 22:48:13.737829 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.737693 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 22:48:13.750914 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.750887 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59"] Jul 10 22:48:13.849440 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.849402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.849638 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.849446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.849638 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.849481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60405b33-0de7-4962-aa44-569185c98026-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.849638 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.849505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/60405b33-0de7-4962-aa44-569185c98026-kube-api-access-v5lfw\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.849638 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.849551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60405b33-0de7-4962-aa44-569185c98026-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.849638 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.849581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60405b33-0de7-4962-aa44-569185c98026-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.849880 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.849658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.950369 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.950287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.950369 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.950322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.950579 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.950425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60405b33-0de7-4962-aa44-569185c98026-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.950579 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.950443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/60405b33-0de7-4962-aa44-569185c98026-kube-api-access-v5lfw\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.950579 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.950477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60405b33-0de7-4962-aa44-569185c98026-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.950579 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.950500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60405b33-0de7-4962-aa44-569185c98026-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.950759 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.950628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.951174 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.951129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60405b33-0de7-4962-aa44-569185c98026-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.952697 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.952669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60405b33-0de7-4962-aa44-569185c98026-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.952858 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.952838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.952919 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.952894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.952919 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.952904 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60405b33-0de7-4962-aa44-569185c98026-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.961588 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.961565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60405b33-0de7-4962-aa44-569185c98026-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.961717 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.961694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/60405b33-0de7-4962-aa44-569185c98026-kube-api-access-v5lfw\") pod \"istiod-openshift-gateway-94bb8fbfd-v2g59\" (UID: \"60405b33-0de7-4962-aa44-569185c98026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:13.969337 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:13.969315 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c7f744b8d-xlkc4" Jul 10 22:48:14.006084 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:14.006054 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:14.161458 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:14.161419 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59"] Jul 10 22:48:14.166282 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:48:14.166250 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60405b33_0de7_4962_aa44_569185c98026.slice/crio-23638659a59942dbf598230357e0c41d88c910f3018bd5d23c7424c5dd74a5b9 WatchSource:0}: Error finding container 23638659a59942dbf598230357e0c41d88c910f3018bd5d23c7424c5dd74a5b9: Status 404 returned error can't find the container with id 23638659a59942dbf598230357e0c41d88c910f3018bd5d23c7424c5dd74a5b9 Jul 10 22:48:14.168300 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:14.168284 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:48:15.035287 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:15.035242 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" event={"ID":"60405b33-0de7-4962-aa44-569185c98026","Type":"ContainerStarted","Data":"23638659a59942dbf598230357e0c41d88c910f3018bd5d23c7424c5dd74a5b9"} Jul 10 22:48:16.508874 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:16.508838 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 22:48:16.509185 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:16.508900 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 22:48:17.047844 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:17.047808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" event={"ID":"60405b33-0de7-4962-aa44-569185c98026","Type":"ContainerStarted","Data":"a2c2ff76fd47c291cde60a16073b5bff32545b44da8997ceff0fe9a5dbe07161"} Jul 10 22:48:17.048048 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:17.048007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:48:17.049832 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:17.049800 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-v2g59 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 22:48:17.049957 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:17.049852 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" podUID="60405b33-0de7-4962-aa44-569185c98026" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 22:48:17.071715 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:17.071665 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" podStartSLOduration=1.731454829 podStartE2EDuration="4.071648811s" totalImagesPullingTime="2.340193982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:48:17.070237521 +0000 UTC m=+315.947929151" watchObservedRunningTime="2026-07-10 22:48:17.071648811 +0000 UTC m=+315.949340430" Jul 10 22:48:18.052470 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:18.052437 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-v2g59 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 22:48:18.052851 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:18.052490 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" podUID="60405b33-0de7-4962-aa44-569185c98026" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 22:48:21.053393 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:48:21.053353 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-v2g59" Jul 10 22:49:23.243073 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.242989 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r"] Jul 10 22:49:23.246331 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.246315 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.248820 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.248797 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-crb7z\"" Jul 10 22:49:23.248917 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.248797 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 22:49:23.249918 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.249902 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 22:49:23.254590 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.254572 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r"] Jul 10 22:49:23.339758 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.339719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8w7\" (UniqueName: \"kubernetes.io/projected/0323d7bb-cb80-46c2-bd54-60aa9019cb65-kube-api-access-xc8w7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.339924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.339766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.339924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.339844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.344307 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.344283 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv"] Jul 10 22:49:23.347670 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.347654 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.357907 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.357886 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv"] Jul 10 22:49:23.441099 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8w7\" (UniqueName: \"kubernetes.io/projected/0323d7bb-cb80-46c2-bd54-60aa9019cb65-kube-api-access-xc8w7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.441268 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.441268 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.441268 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sckl\" (UniqueName: \"kubernetes.io/projected/4792923f-2516-44df-a253-3e2e083ae637-kube-api-access-4sckl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.441268 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.441449 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.441495 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.441590 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.441576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.449323 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.449292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8w7\" (UniqueName: \"kubernetes.io/projected/0323d7bb-cb80-46c2-bd54-60aa9019cb65-kube-api-access-xc8w7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.542576 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.542500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.542710 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.542579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.542710 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.542609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sckl\" (UniqueName: \"kubernetes.io/projected/4792923f-2516-44df-a253-3e2e083ae637-kube-api-access-4sckl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.542959 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.542932 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.543026 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.542940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.550513 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.550491 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sckl\" (UniqueName: \"kubernetes.io/projected/4792923f-2516-44df-a253-3e2e083ae637-kube-api-access-4sckl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.556419 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.556399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:23.639903 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.639877 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8"] Jul 10 22:49:23.643957 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.643930 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.654179 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.654141 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8"] Jul 10 22:49:23.656844 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.656822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:23.696702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.696671 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r"] Jul 10 22:49:23.700925 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:49:23.700899 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0323d7bb_cb80_46c2_bd54_60aa9019cb65.slice/crio-cc728f40159a07a51866c555995dd0efe40abb6e4bccd7b9160bcd881b2671fe WatchSource:0}: Error finding container cc728f40159a07a51866c555995dd0efe40abb6e4bccd7b9160bcd881b2671fe: Status 404 returned error can't find the container with id cc728f40159a07a51866c555995dd0efe40abb6e4bccd7b9160bcd881b2671fe Jul 10 22:49:23.744913 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.744885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.745063 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.744935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.745127 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.745102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/de5ab973-d07a-4701-805c-b6497935a05d-kube-api-access-58678\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.803032 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.803006 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv"] Jul 10 22:49:23.804771 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:49:23.804746 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4792923f_2516_44df_a253_3e2e083ae637.slice/crio-96f884af02b20c941d799ebffb9b21d72f7131cab155d3091018a9d099764451 WatchSource:0}: Error finding container 96f884af02b20c941d799ebffb9b21d72f7131cab155d3091018a9d099764451: Status 404 returned error can't find the container with id 96f884af02b20c941d799ebffb9b21d72f7131cab155d3091018a9d099764451 Jul 10 22:49:23.845622 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.845598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/de5ab973-d07a-4701-805c-b6497935a05d-kube-api-access-58678\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.845762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.845632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.845762 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.845666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.846054 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.846034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.846054 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.846050 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.853541 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.853523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/de5ab973-d07a-4701-805c-b6497935a05d-kube-api-access-58678\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:23.957892 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:23.957862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:24.045314 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.045281 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p"] Jul 10 22:49:24.048813 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.048786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.064430 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.060601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p"] Jul 10 22:49:24.113460 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.113433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8"] Jul 10 22:49:24.115244 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:49:24.115220 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5ab973_d07a_4701_805c_b6497935a05d.slice/crio-3a93eeca5ff3a781c84c73989deb2782cc6f48d470e88a3a8e18793e9344bf1b WatchSource:0}: Error finding container 3a93eeca5ff3a781c84c73989deb2782cc6f48d470e88a3a8e18793e9344bf1b: Status 404 returned error can't find the container with id 3a93eeca5ff3a781c84c73989deb2782cc6f48d470e88a3a8e18793e9344bf1b Jul 10 22:49:24.147901 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.147878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.148016 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.147913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.148016 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.147960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzjc\" (UniqueName: \"kubernetes.io/projected/feb3383c-ee94-4856-886f-206be299707b-kube-api-access-pjzjc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.249213 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.249183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzjc\" (UniqueName: \"kubernetes.io/projected/feb3383c-ee94-4856-886f-206be299707b-kube-api-access-pjzjc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.249627 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.249235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.249627 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.249261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.249627 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.249530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.249627 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.249612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.257198 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.257176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzjc\" (UniqueName: \"kubernetes.io/projected/feb3383c-ee94-4856-886f-206be299707b-kube-api-access-pjzjc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.362580 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.362506 2577 generic.go:356] "Generic (PLEG): container finished" podID="de5ab973-d07a-4701-805c-b6497935a05d" containerID="7b8df4f815bc42398dbc28603ad01e5ce2e2adf40b89020f4afb69dd2fd6ad45" exitCode=0 Jul 10 22:49:24.362751 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.362592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" event={"ID":"de5ab973-d07a-4701-805c-b6497935a05d","Type":"ContainerDied","Data":"7b8df4f815bc42398dbc28603ad01e5ce2e2adf40b89020f4afb69dd2fd6ad45"} Jul 10 22:49:24.362751 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.362651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" event={"ID":"de5ab973-d07a-4701-805c-b6497935a05d","Type":"ContainerStarted","Data":"3a93eeca5ff3a781c84c73989deb2782cc6f48d470e88a3a8e18793e9344bf1b"} Jul 10 22:49:24.364216 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.364195 2577 generic.go:356] "Generic (PLEG): container finished" podID="4792923f-2516-44df-a253-3e2e083ae637" containerID="182fe4ed0354448009162cd7253286d8e3a7de96090e943d51387476698d28db" exitCode=0 Jul 10 22:49:24.364313 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.364278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" event={"ID":"4792923f-2516-44df-a253-3e2e083ae637","Type":"ContainerDied","Data":"182fe4ed0354448009162cd7253286d8e3a7de96090e943d51387476698d28db"} Jul 10 22:49:24.364371 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.364326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" event={"ID":"4792923f-2516-44df-a253-3e2e083ae637","Type":"ContainerStarted","Data":"96f884af02b20c941d799ebffb9b21d72f7131cab155d3091018a9d099764451"} Jul 10 22:49:24.365810 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.365787 2577 generic.go:356] "Generic (PLEG): container finished" podID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerID="0bde601133bdc1e7030b083c9d2bae0351f458fdd10d764934abf9e9f52b93fd" exitCode=0 Jul 10 22:49:24.365919 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.365841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" event={"ID":"0323d7bb-cb80-46c2-bd54-60aa9019cb65","Type":"ContainerDied","Data":"0bde601133bdc1e7030b083c9d2bae0351f458fdd10d764934abf9e9f52b93fd"} Jul 10 22:49:24.365919 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.365874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" event={"ID":"0323d7bb-cb80-46c2-bd54-60aa9019cb65","Type":"ContainerStarted","Data":"cc728f40159a07a51866c555995dd0efe40abb6e4bccd7b9160bcd881b2671fe"} Jul 10 22:49:24.366554 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.366538 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:24.500026 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:24.499999 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p"] Jul 10 22:49:24.501357 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:49:24.501330 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb3383c_ee94_4856_886f_206be299707b.slice/crio-7ca2bddb6b406234fa45db962f2de1498ca6a779a56176ace21cfedcc3aa32ca WatchSource:0}: Error finding container 7ca2bddb6b406234fa45db962f2de1498ca6a779a56176ace21cfedcc3aa32ca: Status 404 returned error can't find the container with id 7ca2bddb6b406234fa45db962f2de1498ca6a779a56176ace21cfedcc3aa32ca Jul 10 22:49:25.373365 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:25.373328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" event={"ID":"4792923f-2516-44df-a253-3e2e083ae637","Type":"ContainerStarted","Data":"d1b01f305e58f28da531a8e272ee7366a27f0f7edd03443ea145129e8a48282c"} Jul 10 22:49:25.375080 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:25.375055 2577 generic.go:356] "Generic (PLEG): container finished" podID="feb3383c-ee94-4856-886f-206be299707b" containerID="517e7713d6ae0240772686b124caa65cc3f3295a48fed407e64f82bc5ad0e5ad" exitCode=0 Jul 10 22:49:25.375199 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:25.375109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" event={"ID":"feb3383c-ee94-4856-886f-206be299707b","Type":"ContainerDied","Data":"517e7713d6ae0240772686b124caa65cc3f3295a48fed407e64f82bc5ad0e5ad"} Jul 10 22:49:25.375199 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:25.375140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" event={"ID":"feb3383c-ee94-4856-886f-206be299707b","Type":"ContainerStarted","Data":"7ca2bddb6b406234fa45db962f2de1498ca6a779a56176ace21cfedcc3aa32ca"} Jul 10 22:49:26.382584 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.382492 2577 generic.go:356] "Generic (PLEG): container finished" podID="de5ab973-d07a-4701-805c-b6497935a05d" containerID="63744f23d170a6857f0301ca301130613d9aeb7370f2a97c0289960badad24d2" exitCode=0 Jul 10 22:49:26.382584 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.382570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" event={"ID":"de5ab973-d07a-4701-805c-b6497935a05d","Type":"ContainerDied","Data":"63744f23d170a6857f0301ca301130613d9aeb7370f2a97c0289960badad24d2"} Jul 10 22:49:26.384512 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.384489 2577 generic.go:356] "Generic (PLEG): container finished" podID="4792923f-2516-44df-a253-3e2e083ae637" containerID="d1b01f305e58f28da531a8e272ee7366a27f0f7edd03443ea145129e8a48282c" exitCode=0 Jul 10 22:49:26.384594 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.384561 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" event={"ID":"4792923f-2516-44df-a253-3e2e083ae637","Type":"ContainerDied","Data":"d1b01f305e58f28da531a8e272ee7366a27f0f7edd03443ea145129e8a48282c"} Jul 10 22:49:26.386372 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.386351 2577 generic.go:356] "Generic (PLEG): container finished" podID="feb3383c-ee94-4856-886f-206be299707b" containerID="d02888e8c06e85479a9ad348e33b5a4b557ff37fc19a167f45a539153e4cc046" exitCode=0 Jul 10 22:49:26.386480 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.386379 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" event={"ID":"feb3383c-ee94-4856-886f-206be299707b","Type":"ContainerDied","Data":"d02888e8c06e85479a9ad348e33b5a4b557ff37fc19a167f45a539153e4cc046"} Jul 10 22:49:26.388350 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.388327 2577 generic.go:356] "Generic (PLEG): container finished" podID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerID="69c7173493a9f688bb08c8d013aab363d7ef7e6e133e81ecf4ef1430eff589e2" exitCode=0 Jul 10 22:49:26.388417 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:26.388401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" event={"ID":"0323d7bb-cb80-46c2-bd54-60aa9019cb65","Type":"ContainerDied","Data":"69c7173493a9f688bb08c8d013aab363d7ef7e6e133e81ecf4ef1430eff589e2"} Jul 10 22:49:27.395567 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.395532 2577 generic.go:356] "Generic (PLEG): container finished" podID="de5ab973-d07a-4701-805c-b6497935a05d" containerID="3b78b8a26cc791ebbde4f46a084feae37be765835241b7b7fb4035ed44295d1a" exitCode=0 Jul 10 22:49:27.396030 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.395612 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" event={"ID":"de5ab973-d07a-4701-805c-b6497935a05d","Type":"ContainerDied","Data":"3b78b8a26cc791ebbde4f46a084feae37be765835241b7b7fb4035ed44295d1a"} Jul 10 22:49:27.397838 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.397814 2577 generic.go:356] "Generic (PLEG): container finished" podID="4792923f-2516-44df-a253-3e2e083ae637" containerID="bd4a049f7c5d46d44091e06b17b20e0f4c59c6a416989f0ae9f557f6321cfc7c" exitCode=0 Jul 10 22:49:27.397962 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.397884 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" event={"ID":"4792923f-2516-44df-a253-3e2e083ae637","Type":"ContainerDied","Data":"bd4a049f7c5d46d44091e06b17b20e0f4c59c6a416989f0ae9f557f6321cfc7c"} Jul 10 22:49:27.399869 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.399848 2577 generic.go:356] "Generic (PLEG): container finished" podID="feb3383c-ee94-4856-886f-206be299707b" containerID="48fbf9233a62a0db53f7202551377b6c2683ecb83398b8874be07ab0a45706d7" exitCode=0 Jul 10 22:49:27.399953 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.399932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" event={"ID":"feb3383c-ee94-4856-886f-206be299707b","Type":"ContainerDied","Data":"48fbf9233a62a0db53f7202551377b6c2683ecb83398b8874be07ab0a45706d7"} Jul 10 22:49:27.401788 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.401765 2577 generic.go:356] "Generic (PLEG): container finished" podID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerID="5ae31b04e4d1b33b2f39b6f26b1af65cbdfaecb2ccbdcbf8bfd4bf32f56b1692" exitCode=0 Jul 10 22:49:27.401868 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:27.401792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" event={"ID":"0323d7bb-cb80-46c2-bd54-60aa9019cb65","Type":"ContainerDied","Data":"5ae31b04e4d1b33b2f39b6f26b1af65cbdfaecb2ccbdcbf8bfd4bf32f56b1692"} Jul 10 22:49:28.552890 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.552864 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:28.592167 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.592142 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-bundle\") pod \"4792923f-2516-44df-a253-3e2e083ae637\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " Jul 10 22:49:28.592344 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.592281 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-util\") pod \"4792923f-2516-44df-a253-3e2e083ae637\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " Jul 10 22:49:28.592344 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.592318 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sckl\" (UniqueName: \"kubernetes.io/projected/4792923f-2516-44df-a253-3e2e083ae637-kube-api-access-4sckl\") pod \"4792923f-2516-44df-a253-3e2e083ae637\" (UID: \"4792923f-2516-44df-a253-3e2e083ae637\") " Jul 10 22:49:28.592932 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.592863 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-bundle" (OuterVolumeSpecName: "bundle") pod "4792923f-2516-44df-a253-3e2e083ae637" (UID: "4792923f-2516-44df-a253-3e2e083ae637"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.595528 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.595506 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4792923f-2516-44df-a253-3e2e083ae637-kube-api-access-4sckl" (OuterVolumeSpecName: "kube-api-access-4sckl") pod "4792923f-2516-44df-a253-3e2e083ae637" (UID: "4792923f-2516-44df-a253-3e2e083ae637"). InnerVolumeSpecName "kube-api-access-4sckl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:49:28.597544 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.597527 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:28.598537 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.598507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-util" (OuterVolumeSpecName: "util") pod "4792923f-2516-44df-a253-3e2e083ae637" (UID: "4792923f-2516-44df-a253-3e2e083ae637"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.599549 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.599527 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:28.620480 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.620456 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:28.693224 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693151 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc8w7\" (UniqueName: \"kubernetes.io/projected/0323d7bb-cb80-46c2-bd54-60aa9019cb65-kube-api-access-xc8w7\") pod \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " Jul 10 22:49:28.693224 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693183 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/de5ab973-d07a-4701-805c-b6497935a05d-kube-api-access-58678\") pod \"de5ab973-d07a-4701-805c-b6497935a05d\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " Jul 10 22:49:28.693224 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693216 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-util\") pod \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " Jul 10 22:49:28.693479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693236 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-bundle\") pod \"de5ab973-d07a-4701-805c-b6497935a05d\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " Jul 10 22:49:28.693479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693282 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-util\") pod \"de5ab973-d07a-4701-805c-b6497935a05d\" (UID: \"de5ab973-d07a-4701-805c-b6497935a05d\") " Jul 10 22:49:28.693479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693299 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-util\") pod \"feb3383c-ee94-4856-886f-206be299707b\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " Jul 10 22:49:28.693479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693344 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-bundle\") pod \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\" (UID: \"0323d7bb-cb80-46c2-bd54-60aa9019cb65\") " Jul 10 22:49:28.693479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693390 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjzjc\" (UniqueName: \"kubernetes.io/projected/feb3383c-ee94-4856-886f-206be299707b-kube-api-access-pjzjc\") pod \"feb3383c-ee94-4856-886f-206be299707b\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " Jul 10 22:49:28.693479 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693421 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-bundle\") pod \"feb3383c-ee94-4856-886f-206be299707b\" (UID: \"feb3383c-ee94-4856-886f-206be299707b\") " Jul 10 22:49:28.693775 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693686 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-util\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.693775 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693705 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sckl\" (UniqueName: \"kubernetes.io/projected/4792923f-2516-44df-a253-3e2e083ae637-kube-api-access-4sckl\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.693775 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.693721 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4792923f-2516-44df-a253-3e2e083ae637-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.694164 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.694129 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-bundle" (OuterVolumeSpecName: "bundle") pod "0323d7bb-cb80-46c2-bd54-60aa9019cb65" (UID: "0323d7bb-cb80-46c2-bd54-60aa9019cb65"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.694449 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.694414 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-bundle" (OuterVolumeSpecName: "bundle") pod "feb3383c-ee94-4856-886f-206be299707b" (UID: "feb3383c-ee94-4856-886f-206be299707b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.694718 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.694679 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-bundle" (OuterVolumeSpecName: "bundle") pod "de5ab973-d07a-4701-805c-b6497935a05d" (UID: "de5ab973-d07a-4701-805c-b6497935a05d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.695782 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.695755 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5ab973-d07a-4701-805c-b6497935a05d-kube-api-access-58678" (OuterVolumeSpecName: "kube-api-access-58678") pod "de5ab973-d07a-4701-805c-b6497935a05d" (UID: "de5ab973-d07a-4701-805c-b6497935a05d"). InnerVolumeSpecName "kube-api-access-58678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:49:28.695889 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.695785 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0323d7bb-cb80-46c2-bd54-60aa9019cb65-kube-api-access-xc8w7" (OuterVolumeSpecName: "kube-api-access-xc8w7") pod "0323d7bb-cb80-46c2-bd54-60aa9019cb65" (UID: "0323d7bb-cb80-46c2-bd54-60aa9019cb65"). InnerVolumeSpecName "kube-api-access-xc8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:49:28.696097 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.696066 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb3383c-ee94-4856-886f-206be299707b-kube-api-access-pjzjc" (OuterVolumeSpecName: "kube-api-access-pjzjc") pod "feb3383c-ee94-4856-886f-206be299707b" (UID: "feb3383c-ee94-4856-886f-206be299707b"). InnerVolumeSpecName "kube-api-access-pjzjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:49:28.699002 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.698959 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-util" (OuterVolumeSpecName: "util") pod "de5ab973-d07a-4701-805c-b6497935a05d" (UID: "de5ab973-d07a-4701-805c-b6497935a05d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.699565 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.699538 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-util" (OuterVolumeSpecName: "util") pod "feb3383c-ee94-4856-886f-206be299707b" (UID: "feb3383c-ee94-4856-886f-206be299707b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.700381 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.700362 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-util" (OuterVolumeSpecName: "util") pod "0323d7bb-cb80-46c2-bd54-60aa9019cb65" (UID: "0323d7bb-cb80-46c2-bd54-60aa9019cb65"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:49:28.794601 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794571 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc8w7\" (UniqueName: \"kubernetes.io/projected/0323d7bb-cb80-46c2-bd54-60aa9019cb65-kube-api-access-xc8w7\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794601 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794597 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/de5ab973-d07a-4701-805c-b6497935a05d-kube-api-access-58678\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794601 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794607 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-util\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794618 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794626 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5ab973-d07a-4701-805c-b6497935a05d-util\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794633 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-util\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794640 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0323d7bb-cb80-46c2-bd54-60aa9019cb65-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794648 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzjc\" (UniqueName: \"kubernetes.io/projected/feb3383c-ee94-4856-886f-206be299707b-kube-api-access-pjzjc\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:28.794806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:28.794656 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb3383c-ee94-4856-886f-206be299707b-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:49:29.413898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.413870 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" Jul 10 22:49:29.414068 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.413893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r" event={"ID":"0323d7bb-cb80-46c2-bd54-60aa9019cb65","Type":"ContainerDied","Data":"cc728f40159a07a51866c555995dd0efe40abb6e4bccd7b9160bcd881b2671fe"} Jul 10 22:49:29.414068 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.413950 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc728f40159a07a51866c555995dd0efe40abb6e4bccd7b9160bcd881b2671fe" Jul 10 22:49:29.416008 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.415988 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" Jul 10 22:49:29.416008 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.416001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8" event={"ID":"de5ab973-d07a-4701-805c-b6497935a05d","Type":"ContainerDied","Data":"3a93eeca5ff3a781c84c73989deb2782cc6f48d470e88a3a8e18793e9344bf1b"} Jul 10 22:49:29.416174 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.416034 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a93eeca5ff3a781c84c73989deb2782cc6f48d470e88a3a8e18793e9344bf1b" Jul 10 22:49:29.418053 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.418037 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" Jul 10 22:49:29.418203 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.418035 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv" event={"ID":"4792923f-2516-44df-a253-3e2e083ae637","Type":"ContainerDied","Data":"96f884af02b20c941d799ebffb9b21d72f7131cab155d3091018a9d099764451"} Jul 10 22:49:29.418203 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.418142 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f884af02b20c941d799ebffb9b21d72f7131cab155d3091018a9d099764451" Jul 10 22:49:29.420157 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.420136 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" event={"ID":"feb3383c-ee94-4856-886f-206be299707b","Type":"ContainerDied","Data":"7ca2bddb6b406234fa45db962f2de1498ca6a779a56176ace21cfedcc3aa32ca"} Jul 10 22:49:29.420249 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.420171 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca2bddb6b406234fa45db962f2de1498ca6a779a56176ace21cfedcc3aa32ca" Jul 10 22:49:29.420249 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:29.420177 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p" Jul 10 22:49:44.436269 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436234 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs"] Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436540 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5ab973-d07a-4701-805c-b6497935a05d" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436551 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5ab973-d07a-4701-805c-b6497935a05d" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436567 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb3383c-ee94-4856-886f-206be299707b" containerName="pull" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436572 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb3383c-ee94-4856-886f-206be299707b" containerName="pull" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436580 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436585 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436592 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb3383c-ee94-4856-886f-206be299707b" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436598 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb3383c-ee94-4856-886f-206be299707b" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436605 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5ab973-d07a-4701-805c-b6497935a05d" containerName="extract" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436610 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5ab973-d07a-4701-805c-b6497935a05d" containerName="extract" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436618 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4792923f-2516-44df-a253-3e2e083ae637" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436623 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4792923f-2516-44df-a253-3e2e083ae637" containerName="util" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436630 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4792923f-2516-44df-a253-3e2e083ae637" containerName="extract" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436635 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4792923f-2516-44df-a253-3e2e083ae637" containerName="extract" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436641 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5ab973-d07a-4701-805c-b6497935a05d" containerName="pull" Jul 10 22:49:44.436641 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436647 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5ab973-d07a-4701-805c-b6497935a05d" containerName="pull" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436653 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4792923f-2516-44df-a253-3e2e083ae637" containerName="pull" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436658 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4792923f-2516-44df-a253-3e2e083ae637" containerName="pull" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436664 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerName="pull" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436669 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerName="pull" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436674 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb3383c-ee94-4856-886f-206be299707b" containerName="extract" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436679 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb3383c-ee94-4856-886f-206be299707b" containerName="extract" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436686 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerName="extract" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436692 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerName="extract" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436739 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb3383c-ee94-4856-886f-206be299707b" containerName="extract" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436747 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0323d7bb-cb80-46c2-bd54-60aa9019cb65" containerName="extract" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436755 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4792923f-2516-44df-a253-3e2e083ae637" containerName="extract" Jul 10 22:49:44.437144 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.436761 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5ab973-d07a-4701-805c-b6497935a05d" containerName="extract" Jul 10 22:49:44.441009 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.440992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" Jul 10 22:49:44.445545 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.445520 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4p67z\"" Jul 10 22:49:44.445759 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.445741 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 22:49:44.446650 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.446631 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 22:49:44.446650 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.446645 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 22:49:44.453836 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.453815 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs"] Jul 10 22:49:44.539221 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.539183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bmb\" (UniqueName: \"kubernetes.io/projected/030ba25a-8e24-4ad7-ae36-68db15524ea0-kube-api-access-d2bmb\") pod \"dns-operator-controller-manager-864dd78c4c-fkmbs\" (UID: \"030ba25a-8e24-4ad7-ae36-68db15524ea0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" Jul 10 22:49:44.640184 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.640148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bmb\" (UniqueName: \"kubernetes.io/projected/030ba25a-8e24-4ad7-ae36-68db15524ea0-kube-api-access-d2bmb\") pod \"dns-operator-controller-manager-864dd78c4c-fkmbs\" (UID: \"030ba25a-8e24-4ad7-ae36-68db15524ea0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" Jul 10 22:49:44.648349 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.648324 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bmb\" (UniqueName: \"kubernetes.io/projected/030ba25a-8e24-4ad7-ae36-68db15524ea0-kube-api-access-d2bmb\") pod \"dns-operator-controller-manager-864dd78c4c-fkmbs\" (UID: \"030ba25a-8e24-4ad7-ae36-68db15524ea0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" Jul 10 22:49:44.751856 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.751773 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" Jul 10 22:49:44.884893 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:44.884870 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs"] Jul 10 22:49:44.887042 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:49:44.887002 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030ba25a_8e24_4ad7_ae36_68db15524ea0.slice/crio-6eba725d34c643bb6920843c98bb6187b68f58ab23285b84fbc03154bd2f8022 WatchSource:0}: Error finding container 6eba725d34c643bb6920843c98bb6187b68f58ab23285b84fbc03154bd2f8022: Status 404 returned error can't find the container with id 6eba725d34c643bb6920843c98bb6187b68f58ab23285b84fbc03154bd2f8022 Jul 10 22:49:45.497543 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:45.497511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" event={"ID":"030ba25a-8e24-4ad7-ae36-68db15524ea0","Type":"ContainerStarted","Data":"6eba725d34c643bb6920843c98bb6187b68f58ab23285b84fbc03154bd2f8022"} Jul 10 22:49:47.509620 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:47.509580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" event={"ID":"030ba25a-8e24-4ad7-ae36-68db15524ea0","Type":"ContainerStarted","Data":"5e635f20d8dc6f4ee40060568580a45736ccda94a8c09d98e01769313457d9eb"} Jul 10 22:49:47.510010 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:47.509674 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" Jul 10 22:49:47.528807 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:47.528716 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" podStartSLOduration=1.149891069 podStartE2EDuration="3.528700986s" totalImagesPullingTime="2.378809917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:49:47.526754564 +0000 UTC m=+406.404446205" watchObservedRunningTime="2026-07-10 22:49:47.528700986 +0000 UTC m=+406.406392635" Jul 10 22:49:51.740059 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.740027 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff4dd9b5c-qjknf"] Jul 10 22:49:51.743411 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.743390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:51.770327 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.770289 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff4dd9b5c-qjknf"] Jul 10 22:49:51.904291 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.904251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-trusted-ca-bundle\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:51.904465 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.904316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f044aa54-b6cd-4d3c-998b-095c77297600-console-serving-cert\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:51.904465 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.904340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f044aa54-b6cd-4d3c-998b-095c77297600-console-oauth-config\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:51.904465 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.904370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-console-config\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:51.904465 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.904388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-oauth-serving-cert\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:51.904465 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.904433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv28s\" (UniqueName: \"kubernetes.io/projected/f044aa54-b6cd-4d3c-998b-095c77297600-kube-api-access-vv28s\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:51.904465 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:51.904460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-service-ca\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.005282 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.005193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f044aa54-b6cd-4d3c-998b-095c77297600-console-serving-cert\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.005282 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.005229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f044aa54-b6cd-4d3c-998b-095c77297600-console-oauth-config\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.005282 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.005252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-console-config\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.005538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.005304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-oauth-serving-cert\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.005538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.005365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv28s\" (UniqueName: \"kubernetes.io/projected/f044aa54-b6cd-4d3c-998b-095c77297600-kube-api-access-vv28s\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.005538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.005404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-service-ca\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.005538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.005490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-trusted-ca-bundle\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.006175 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.006146 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-console-config\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.006310 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.006175 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-oauth-serving-cert\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.006310 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.006181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-service-ca\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.006406 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.006345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f044aa54-b6cd-4d3c-998b-095c77297600-trusted-ca-bundle\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.007801 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.007781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f044aa54-b6cd-4d3c-998b-095c77297600-console-oauth-config\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.007895 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.007825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f044aa54-b6cd-4d3c-998b-095c77297600-console-serving-cert\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.013530 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.013512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv28s\" (UniqueName: \"kubernetes.io/projected/f044aa54-b6cd-4d3c-998b-095c77297600-kube-api-access-vv28s\") pod \"console-5ff4dd9b5c-qjknf\" (UID: \"f044aa54-b6cd-4d3c-998b-095c77297600\") " pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.053081 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.053043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:49:52.194891 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.194864 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff4dd9b5c-qjknf"] Jul 10 22:49:52.196198 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:49:52.196172 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf044aa54_b6cd_4d3c_998b_095c77297600.slice/crio-337e13baee2cc79f3b550bcb2cc7ef1f33fce91ff27e5e81a1c2133c1a73a1b4 WatchSource:0}: Error finding container 337e13baee2cc79f3b550bcb2cc7ef1f33fce91ff27e5e81a1c2133c1a73a1b4: Status 404 returned error can't find the container with id 337e13baee2cc79f3b550bcb2cc7ef1f33fce91ff27e5e81a1c2133c1a73a1b4 Jul 10 22:49:52.535706 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.535670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff4dd9b5c-qjknf" event={"ID":"f044aa54-b6cd-4d3c-998b-095c77297600","Type":"ContainerStarted","Data":"802ba84cf817036d3219927a16981dc690449fb7938c6e32a7f7fe2fcbaf4630"} Jul 10 22:49:52.535866 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.535722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff4dd9b5c-qjknf" event={"ID":"f044aa54-b6cd-4d3c-998b-095c77297600","Type":"ContainerStarted","Data":"337e13baee2cc79f3b550bcb2cc7ef1f33fce91ff27e5e81a1c2133c1a73a1b4"} Jul 10 22:49:52.562721 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:52.562663 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff4dd9b5c-qjknf" podStartSLOduration=1.562646727 podStartE2EDuration="1.562646727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:49:52.561306838 +0000 UTC m=+411.438998500" watchObservedRunningTime="2026-07-10 22:49:52.562646727 +0000 UTC m=+411.440338428" Jul 10 22:49:58.517576 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:49:58.517545 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fkmbs" Jul 10 22:50:00.962535 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:00.962497 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf"] Jul 10 22:50:00.967069 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:00.967052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:00.970225 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:00.970202 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tj8sl\"" Jul 10 22:50:00.977890 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:00.977867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchbz\" (UniqueName: \"kubernetes.io/projected/1e8b2db8-afce-429c-8478-2ab5c76b923c-kube-api-access-dchbz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:00.977999 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:00.977927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2db8-afce-429c-8478-2ab5c76b923c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:00.991472 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:00.991445 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf"] Jul 10 22:50:01.078876 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.078841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchbz\" (UniqueName: \"kubernetes.io/projected/1e8b2db8-afce-429c-8478-2ab5c76b923c-kube-api-access-dchbz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:01.079078 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.078899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2db8-afce-429c-8478-2ab5c76b923c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:01.079298 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.079279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2db8-afce-429c-8478-2ab5c76b923c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:01.097254 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.097231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchbz\" (UniqueName: \"kubernetes.io/projected/1e8b2db8-afce-429c-8478-2ab5c76b923c-kube-api-access-dchbz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:01.102614 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.102585 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf"] Jul 10 22:50:01.102922 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.102906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:01.240569 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.240545 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf"] Jul 10 22:50:01.243043 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:50:01.243014 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8b2db8_afce_429c_8478_2ab5c76b923c.slice/crio-a1e8a39332162f97be72528507db83fdf0e88e6b81f794623711f493c1f8d1a5 WatchSource:0}: Error finding container a1e8a39332162f97be72528507db83fdf0e88e6b81f794623711f493c1f8d1a5: Status 404 returned error can't find the container with id a1e8a39332162f97be72528507db83fdf0e88e6b81f794623711f493c1f8d1a5 Jul 10 22:50:01.376507 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.376469 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf"] Jul 10 22:50:01.387368 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.387338 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf"] Jul 10 22:50:01.429283 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.429252 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr"] Jul 10 22:50:01.433888 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.433866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.449280 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.449248 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr"] Jul 10 22:50:01.482372 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.482335 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40c3d3be-4add-45c3-901a-4c4028ce341c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdnrr\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.482554 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.482451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/40c3d3be-4add-45c3-901a-4c4028ce341c-kube-api-access-crhk4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdnrr\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.583689 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.583594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/40c3d3be-4add-45c3-901a-4c4028ce341c-kube-api-access-crhk4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdnrr\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.583689 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.583676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40c3d3be-4add-45c3-901a-4c4028ce341c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdnrr\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.584084 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.584064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40c3d3be-4add-45c3-901a-4c4028ce341c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdnrr\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.591577 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.591550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/40c3d3be-4add-45c3-901a-4c4028ce341c-kube-api-access-crhk4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cdnrr\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.745682 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.745648 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:01.881821 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:01.881794 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr"] Jul 10 22:50:01.883507 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:50:01.883474 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c3d3be_4add_45c3_901a_4c4028ce341c.slice/crio-886de3ef05dfe38d7f751a76e9feea278e39be41ee3479386b146853e446e14a WatchSource:0}: Error finding container 886de3ef05dfe38d7f751a76e9feea278e39be41ee3479386b146853e446e14a: Status 404 returned error can't find the container with id 886de3ef05dfe38d7f751a76e9feea278e39be41ee3479386b146853e446e14a Jul 10 22:50:02.053768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:02.053737 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:50:02.054145 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:02.053829 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:50:02.058563 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:02.058528 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:50:02.590955 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:02.590906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" event={"ID":"40c3d3be-4add-45c3-901a-4c4028ce341c","Type":"ContainerStarted","Data":"886de3ef05dfe38d7f751a76e9feea278e39be41ee3479386b146853e446e14a"} Jul 10 22:50:02.595529 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:02.595499 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff4dd9b5c-qjknf" Jul 10 22:50:02.660364 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:02.660327 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-876b77c67-k6542"] Jul 10 22:50:06.628695 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.628656 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" podUID="1e8b2db8-afce-429c-8478-2ab5c76b923c" containerName="manager" containerID="cri-o://0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5" gracePeriod=2 Jul 10 22:50:06.630458 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.630424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" event={"ID":"40c3d3be-4add-45c3-901a-4c4028ce341c","Type":"ContainerStarted","Data":"7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2"} Jul 10 22:50:06.630583 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.630516 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:06.654252 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.654203 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" podStartSLOduration=1.453498138 podStartE2EDuration="5.654187595s" totalImagesPullingTime="4.200689457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:50:06.653615726 +0000 UTC m=+425.531307345" watchObservedRunningTime="2026-07-10 22:50:06.654187595 +0000 UTC m=+425.531879213" Jul 10 22:50:06.891620 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.891559 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:06.893896 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:50:06.893872 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" is forbidden: User \"system:node:ip-10-0-139-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-196.ec2.internal' and this object" podUID="1e8b2db8-afce-429c-8478-2ab5c76b923c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:06.931447 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.931421 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2db8-afce-429c-8478-2ab5c76b923c-extensions-socket-volume\") pod \"1e8b2db8-afce-429c-8478-2ab5c76b923c\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " Jul 10 22:50:06.931614 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.931457 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dchbz\" (UniqueName: \"kubernetes.io/projected/1e8b2db8-afce-429c-8478-2ab5c76b923c-kube-api-access-dchbz\") pod \"1e8b2db8-afce-429c-8478-2ab5c76b923c\" (UID: \"1e8b2db8-afce-429c-8478-2ab5c76b923c\") " Jul 10 22:50:06.931760 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.931733 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8b2db8-afce-429c-8478-2ab5c76b923c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1e8b2db8-afce-429c-8478-2ab5c76b923c" (UID: "1e8b2db8-afce-429c-8478-2ab5c76b923c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:50:06.933508 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:06.933487 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8b2db8-afce-429c-8478-2ab5c76b923c-kube-api-access-dchbz" (OuterVolumeSpecName: "kube-api-access-dchbz") pod "1e8b2db8-afce-429c-8478-2ab5c76b923c" (UID: "1e8b2db8-afce-429c-8478-2ab5c76b923c"). InnerVolumeSpecName "kube-api-access-dchbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:50:07.032337 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.032294 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2db8-afce-429c-8478-2ab5c76b923c-extensions-socket-volume\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:07.032337 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.032329 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchbz\" (UniqueName: \"kubernetes.io/projected/1e8b2db8-afce-429c-8478-2ab5c76b923c-kube-api-access-dchbz\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:07.635496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.635464 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8b2db8-afce-429c-8478-2ab5c76b923c" path="/var/lib/kubelet/pods/1e8b2db8-afce-429c-8478-2ab5c76b923c/volumes" Jul 10 22:50:07.636959 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.636930 2577 generic.go:356] "Generic (PLEG): container finished" podID="1e8b2db8-afce-429c-8478-2ab5c76b923c" containerID="0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5" exitCode=2 Jul 10 22:50:07.637125 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.636986 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:07.637125 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.637011 2577 scope.go:117] "RemoveContainer" containerID="0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5" Jul 10 22:50:07.646703 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.646681 2577 scope.go:117] "RemoveContainer" containerID="0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5" Jul 10 22:50:07.647044 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:50:07.647020 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5\": container with ID starting with 0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5 not found: ID does not exist" containerID="0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5" Jul 10 22:50:07.647123 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:07.647052 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5"} err="failed to get container status \"0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5\": rpc error: code = NotFound desc = could not find container \"0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5\": container with ID starting with 0fa94767cf7282403e60ac39d5a042ea535c1594530fe1ab0843bbb7f7dbc1e5 not found: ID does not exist" Jul 10 22:50:07.650205 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:50:07.650180 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4tmrf\" is forbidden: User \"system:node:ip-10-0-139-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-196.ec2.internal' and this object" podUID="1e8b2db8-afce-429c-8478-2ab5c76b923c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4tmrf" Jul 10 22:50:17.395181 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.395137 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr"] Jul 10 22:50:17.395649 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.395467 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" podUID="40c3d3be-4add-45c3-901a-4c4028ce341c" containerName="manager" containerID="cri-o://7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2" gracePeriod=10 Jul 10 22:50:17.397092 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.397070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:17.655861 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.655834 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:17.686609 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.686581 2577 generic.go:356] "Generic (PLEG): container finished" podID="40c3d3be-4add-45c3-901a-4c4028ce341c" containerID="7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2" exitCode=0 Jul 10 22:50:17.686790 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.686638 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" Jul 10 22:50:17.686790 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.686673 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" event={"ID":"40c3d3be-4add-45c3-901a-4c4028ce341c","Type":"ContainerDied","Data":"7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2"} Jul 10 22:50:17.686790 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.686726 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr" event={"ID":"40c3d3be-4add-45c3-901a-4c4028ce341c","Type":"ContainerDied","Data":"886de3ef05dfe38d7f751a76e9feea278e39be41ee3479386b146853e446e14a"} Jul 10 22:50:17.686790 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.686753 2577 scope.go:117] "RemoveContainer" containerID="7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2" Jul 10 22:50:17.695906 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.695882 2577 scope.go:117] "RemoveContainer" containerID="7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2" Jul 10 22:50:17.696227 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:50:17.696208 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2\": container with ID starting with 7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2 not found: ID does not exist" containerID="7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2" Jul 10 22:50:17.696300 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.696236 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2"} err="failed to get container status \"7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2\": rpc error: code = NotFound desc = could not find container \"7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2\": container with ID starting with 7dc4207b23b84696ca86e981eef99afe56343bc3966afa554238a5272eacc1f2 not found: ID does not exist" Jul 10 22:50:17.712578 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.712553 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6"] Jul 10 22:50:17.712878 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.712867 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8b2db8-afce-429c-8478-2ab5c76b923c" containerName="manager" Jul 10 22:50:17.712924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.712880 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8b2db8-afce-429c-8478-2ab5c76b923c" containerName="manager" Jul 10 22:50:17.712924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.712904 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c3d3be-4add-45c3-901a-4c4028ce341c" containerName="manager" Jul 10 22:50:17.712924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.712910 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c3d3be-4add-45c3-901a-4c4028ce341c" containerName="manager" Jul 10 22:50:17.713025 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.712955 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c3d3be-4add-45c3-901a-4c4028ce341c" containerName="manager" Jul 10 22:50:17.713025 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.712965 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8b2db8-afce-429c-8478-2ab5c76b923c" containerName="manager" Jul 10 22:50:17.717151 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.717136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:17.719466 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.719443 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40c3d3be-4add-45c3-901a-4c4028ce341c-extensions-socket-volume\") pod \"40c3d3be-4add-45c3-901a-4c4028ce341c\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " Jul 10 22:50:17.719613 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.719485 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/40c3d3be-4add-45c3-901a-4c4028ce341c-kube-api-access-crhk4\") pod \"40c3d3be-4add-45c3-901a-4c4028ce341c\" (UID: \"40c3d3be-4add-45c3-901a-4c4028ce341c\") " Jul 10 22:50:17.720033 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.720006 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c3d3be-4add-45c3-901a-4c4028ce341c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "40c3d3be-4add-45c3-901a-4c4028ce341c" (UID: "40c3d3be-4add-45c3-901a-4c4028ce341c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:50:17.721770 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.721748 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c3d3be-4add-45c3-901a-4c4028ce341c-kube-api-access-crhk4" (OuterVolumeSpecName: "kube-api-access-crhk4") pod "40c3d3be-4add-45c3-901a-4c4028ce341c" (UID: "40c3d3be-4add-45c3-901a-4c4028ce341c"). InnerVolumeSpecName "kube-api-access-crhk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:50:17.727507 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.727485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6"] Jul 10 22:50:17.820646 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.820611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53885095-5123-468e-b4f0-e94aea3c2189-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-85qc6\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:17.820811 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.820670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt75d\" (UniqueName: \"kubernetes.io/projected/53885095-5123-468e-b4f0-e94aea3c2189-kube-api-access-pt75d\") pod \"kuadrant-operator-controller-manager-86878c769f-85qc6\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:17.820811 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.820788 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40c3d3be-4add-45c3-901a-4c4028ce341c-extensions-socket-volume\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:17.820811 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.820810 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/40c3d3be-4add-45c3-901a-4c4028ce341c-kube-api-access-crhk4\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:17.921565 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.921481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53885095-5123-468e-b4f0-e94aea3c2189-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-85qc6\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:17.921565 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.921536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt75d\" (UniqueName: \"kubernetes.io/projected/53885095-5123-468e-b4f0-e94aea3c2189-kube-api-access-pt75d\") pod \"kuadrant-operator-controller-manager-86878c769f-85qc6\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:17.921897 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.921876 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53885095-5123-468e-b4f0-e94aea3c2189-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-85qc6\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:17.929909 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:17.929880 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt75d\" (UniqueName: \"kubernetes.io/projected/53885095-5123-468e-b4f0-e94aea3c2189-kube-api-access-pt75d\") pod \"kuadrant-operator-controller-manager-86878c769f-85qc6\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:18.010372 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.010335 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr"] Jul 10 22:50:18.016502 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.016476 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cdnrr"] Jul 10 22:50:18.036015 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.035991 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:18.173024 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.173002 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6"] Jul 10 22:50:18.175142 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:50:18.175119 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53885095_5123_468e_b4f0_e94aea3c2189.slice/crio-a8dff176a7112cce10b93d58276d74aa6d4367bd23fcfc7625617688e30cb803 WatchSource:0}: Error finding container a8dff176a7112cce10b93d58276d74aa6d4367bd23fcfc7625617688e30cb803: Status 404 returned error can't find the container with id a8dff176a7112cce10b93d58276d74aa6d4367bd23fcfc7625617688e30cb803 Jul 10 22:50:18.693439 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.693405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" event={"ID":"53885095-5123-468e-b4f0-e94aea3c2189","Type":"ContainerStarted","Data":"006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456"} Jul 10 22:50:18.693878 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.693457 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" event={"ID":"53885095-5123-468e-b4f0-e94aea3c2189","Type":"ContainerStarted","Data":"a8dff176a7112cce10b93d58276d74aa6d4367bd23fcfc7625617688e30cb803"} Jul 10 22:50:18.693878 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.693503 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:50:18.713860 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:18.713805 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" podStartSLOduration=1.713782711 podStartE2EDuration="1.713782711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:50:18.712150228 +0000 UTC m=+437.589841865" watchObservedRunningTime="2026-07-10 22:50:18.713782711 +0000 UTC m=+437.591474330" Jul 10 22:50:19.635667 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:19.635633 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c3d3be-4add-45c3-901a-4c4028ce341c" path="/var/lib/kubelet/pods/40c3d3be-4add-45c3-901a-4c4028ce341c/volumes" Jul 10 22:50:27.692664 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:27.692628 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-876b77c67-k6542" podUID="1000010c-cb01-4992-b71f-d0c09728da92" containerName="console" containerID="cri-o://2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9" gracePeriod=15 Jul 10 22:50:27.955061 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:27.955000 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:50:28.014934 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.014899 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqnhp\" (UniqueName: \"kubernetes.io/projected/1000010c-cb01-4992-b71f-d0c09728da92-kube-api-access-vqnhp\") pod \"1000010c-cb01-4992-b71f-d0c09728da92\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " Jul 10 22:50:28.015107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.014956 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-serving-cert\") pod \"1000010c-cb01-4992-b71f-d0c09728da92\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " Jul 10 22:50:28.015107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015013 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-service-ca\") pod \"1000010c-cb01-4992-b71f-d0c09728da92\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " Jul 10 22:50:28.015107 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015043 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-console-config\") pod \"1000010c-cb01-4992-b71f-d0c09728da92\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " Jul 10 22:50:28.015275 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015221 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-oauth-config\") pod \"1000010c-cb01-4992-b71f-d0c09728da92\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " Jul 10 22:50:28.015333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015279 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-oauth-serving-cert\") pod \"1000010c-cb01-4992-b71f-d0c09728da92\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " Jul 10 22:50:28.015333 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015303 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-trusted-ca-bundle\") pod \"1000010c-cb01-4992-b71f-d0c09728da92\" (UID: \"1000010c-cb01-4992-b71f-d0c09728da92\") " Jul 10 22:50:28.015492 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015464 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-service-ca" (OuterVolumeSpecName: "service-ca") pod "1000010c-cb01-4992-b71f-d0c09728da92" (UID: "1000010c-cb01-4992-b71f-d0c09728da92"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:50:28.015492 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015471 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-console-config" (OuterVolumeSpecName: "console-config") pod "1000010c-cb01-4992-b71f-d0c09728da92" (UID: "1000010c-cb01-4992-b71f-d0c09728da92"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:50:28.015640 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015622 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:28.015692 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015648 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-console-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:28.015692 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015663 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1000010c-cb01-4992-b71f-d0c09728da92" (UID: "1000010c-cb01-4992-b71f-d0c09728da92"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:50:28.015898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.015877 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1000010c-cb01-4992-b71f-d0c09728da92" (UID: "1000010c-cb01-4992-b71f-d0c09728da92"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:50:28.017285 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.017257 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1000010c-cb01-4992-b71f-d0c09728da92" (UID: "1000010c-cb01-4992-b71f-d0c09728da92"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:50:28.017560 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.017535 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1000010c-cb01-4992-b71f-d0c09728da92" (UID: "1000010c-cb01-4992-b71f-d0c09728da92"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:50:28.017623 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.017589 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1000010c-cb01-4992-b71f-d0c09728da92-kube-api-access-vqnhp" (OuterVolumeSpecName: "kube-api-access-vqnhp") pod "1000010c-cb01-4992-b71f-d0c09728da92" (UID: "1000010c-cb01-4992-b71f-d0c09728da92"). InnerVolumeSpecName "kube-api-access-vqnhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:50:28.116918 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.116883 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-oauth-config\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:28.116918 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.116910 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-oauth-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:28.116918 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.116920 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1000010c-cb01-4992-b71f-d0c09728da92-trusted-ca-bundle\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:28.117186 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.116930 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqnhp\" (UniqueName: \"kubernetes.io/projected/1000010c-cb01-4992-b71f-d0c09728da92-kube-api-access-vqnhp\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:28.117186 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.116938 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1000010c-cb01-4992-b71f-d0c09728da92-console-serving-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:50:28.754553 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.754514 2577 generic.go:356] "Generic (PLEG): container finished" podID="1000010c-cb01-4992-b71f-d0c09728da92" containerID="2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9" exitCode=0 Jul 10 22:50:28.754958 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.754568 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-876b77c67-k6542" Jul 10 22:50:28.754958 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.754597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-876b77c67-k6542" event={"ID":"1000010c-cb01-4992-b71f-d0c09728da92","Type":"ContainerDied","Data":"2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9"} Jul 10 22:50:28.754958 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.754671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-876b77c67-k6542" event={"ID":"1000010c-cb01-4992-b71f-d0c09728da92","Type":"ContainerDied","Data":"abc5d051336112cede5851f0d63dcdbbd8597379e7cbd0839d366049834d6bde"} Jul 10 22:50:28.754958 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.754693 2577 scope.go:117] "RemoveContainer" containerID="2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9" Jul 10 22:50:28.764628 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.764608 2577 scope.go:117] "RemoveContainer" containerID="2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9" Jul 10 22:50:28.764892 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:50:28.764874 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9\": container with ID starting with 2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9 not found: ID does not exist" containerID="2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9" Jul 10 22:50:28.764961 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.764901 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9"} err="failed to get container status \"2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9\": rpc error: code = NotFound desc = could not find container \"2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9\": container with ID starting with 2b72677ad29aa0f1be18f071f4f118dea65f8c2ee026cfda20f3817c9c5ffbc9 not found: ID does not exist" Jul 10 22:50:28.777102 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.777070 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-876b77c67-k6542"] Jul 10 22:50:28.779126 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:28.779102 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-876b77c67-k6542"] Jul 10 22:50:29.635057 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:29.635017 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1000010c-cb01-4992-b71f-d0c09728da92" path="/var/lib/kubelet/pods/1000010c-cb01-4992-b71f-d0c09728da92/volumes" Jul 10 22:50:29.701713 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:50:29.701684 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:51:25.258521 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.258486 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-87cdc47f5-2nvpm"] Jul 10 22:51:25.259134 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.259035 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1000010c-cb01-4992-b71f-d0c09728da92" containerName="console" Jul 10 22:51:25.259134 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.259056 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1000010c-cb01-4992-b71f-d0c09728da92" containerName="console" Jul 10 22:51:25.259260 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.259176 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1000010c-cb01-4992-b71f-d0c09728da92" containerName="console" Jul 10 22:51:25.262484 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.262462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.266123 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.266103 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 22:51:25.266123 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.266112 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wj8k9\"" Jul 10 22:51:25.266350 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.266112 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 22:51:25.278918 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.278887 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87cdc47f5-2nvpm"] Jul 10 22:51:25.409702 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.409667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30465094-d587-43de-8fe5-648e5f468ab1-webhook-cert\") pod \"maas-controller-87cdc47f5-2nvpm\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.409893 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.409732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp5s\" (UniqueName: \"kubernetes.io/projected/30465094-d587-43de-8fe5-648e5f468ab1-kube-api-access-zfp5s\") pod \"maas-controller-87cdc47f5-2nvpm\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.416558 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.416529 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-547499fdd5-hwfj6"] Jul 10 22:51:25.419864 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.419845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.429764 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.429716 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-547499fdd5-hwfj6"] Jul 10 22:51:25.510857 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.510773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp5s\" (UniqueName: \"kubernetes.io/projected/30465094-d587-43de-8fe5-648e5f468ab1-kube-api-access-zfp5s\") pod \"maas-controller-87cdc47f5-2nvpm\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.510857 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.510831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fc0d071-494b-47a3-ac10-ba521af363d9-webhook-cert\") pod \"maas-controller-547499fdd5-hwfj6\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.511088 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.510942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30465094-d587-43de-8fe5-648e5f468ab1-webhook-cert\") pod \"maas-controller-87cdc47f5-2nvpm\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.511088 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.511004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6vff\" (UniqueName: \"kubernetes.io/projected/4fc0d071-494b-47a3-ac10-ba521af363d9-kube-api-access-m6vff\") pod \"maas-controller-547499fdd5-hwfj6\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.513486 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.513453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30465094-d587-43de-8fe5-648e5f468ab1-webhook-cert\") pod \"maas-controller-87cdc47f5-2nvpm\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.518703 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.518680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp5s\" (UniqueName: \"kubernetes.io/projected/30465094-d587-43de-8fe5-648e5f468ab1-kube-api-access-zfp5s\") pod \"maas-controller-87cdc47f5-2nvpm\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.572550 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.572514 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:25.611611 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.611575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6vff\" (UniqueName: \"kubernetes.io/projected/4fc0d071-494b-47a3-ac10-ba521af363d9-kube-api-access-m6vff\") pod \"maas-controller-547499fdd5-hwfj6\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.611828 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.611647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fc0d071-494b-47a3-ac10-ba521af363d9-webhook-cert\") pod \"maas-controller-547499fdd5-hwfj6\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.614504 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.614478 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fc0d071-494b-47a3-ac10-ba521af363d9-webhook-cert\") pod \"maas-controller-547499fdd5-hwfj6\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.619908 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.619882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6vff\" (UniqueName: \"kubernetes.io/projected/4fc0d071-494b-47a3-ac10-ba521af363d9-kube-api-access-m6vff\") pod \"maas-controller-547499fdd5-hwfj6\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.706796 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.706774 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87cdc47f5-2nvpm"] Jul 10 22:51:25.709051 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:51:25.709021 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30465094_d587_43de_8fe5_648e5f468ab1.slice/crio-42e575b5da8f0d67d86b6ccbc23da1515400d7205a5228f3822143e72c74ff29 WatchSource:0}: Error finding container 42e575b5da8f0d67d86b6ccbc23da1515400d7205a5228f3822143e72c74ff29: Status 404 returned error can't find the container with id 42e575b5da8f0d67d86b6ccbc23da1515400d7205a5228f3822143e72c74ff29 Jul 10 22:51:25.730804 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.730780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:25.863384 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:25.863357 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-547499fdd5-hwfj6"] Jul 10 22:51:25.864908 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:51:25.864882 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc0d071_494b_47a3_ac10_ba521af363d9.slice/crio-3a06ca9901dc939a988549e2d3730670dc9490a7c7f2e4e7f55a0332a2cc477d WatchSource:0}: Error finding container 3a06ca9901dc939a988549e2d3730670dc9490a7c7f2e4e7f55a0332a2cc477d: Status 404 returned error can't find the container with id 3a06ca9901dc939a988549e2d3730670dc9490a7c7f2e4e7f55a0332a2cc477d Jul 10 22:51:26.062086 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:26.061997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" event={"ID":"30465094-d587-43de-8fe5-648e5f468ab1","Type":"ContainerStarted","Data":"42e575b5da8f0d67d86b6ccbc23da1515400d7205a5228f3822143e72c74ff29"} Jul 10 22:51:26.063217 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:26.063194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-547499fdd5-hwfj6" event={"ID":"4fc0d071-494b-47a3-ac10-ba521af363d9","Type":"ContainerStarted","Data":"3a06ca9901dc939a988549e2d3730670dc9490a7c7f2e4e7f55a0332a2cc477d"} Jul 10 22:51:28.084488 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:28.084379 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" event={"ID":"30465094-d587-43de-8fe5-648e5f468ab1","Type":"ContainerStarted","Data":"de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94"} Jul 10 22:51:28.084488 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:28.084453 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:28.086073 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:28.086045 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-547499fdd5-hwfj6" event={"ID":"4fc0d071-494b-47a3-ac10-ba521af363d9","Type":"ContainerStarted","Data":"1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7"} Jul 10 22:51:28.086199 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:28.086097 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:28.110206 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:28.110160 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" podStartSLOduration=1.068445158 podStartE2EDuration="3.110145886s" totalImagesPullingTime="2.041700728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:51:28.108262491 +0000 UTC m=+506.985954136" watchObservedRunningTime="2026-07-10 22:51:28.110145886 +0000 UTC m=+506.987837504" Jul 10 22:51:28.139080 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:28.139019 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-547499fdd5-hwfj6" podStartSLOduration=1.242332227 podStartE2EDuration="3.139000067s" totalImagesPullingTime="1.89666784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:51:28.135892557 +0000 UTC m=+507.013584176" watchObservedRunningTime="2026-07-10 22:51:28.139000067 +0000 UTC m=+507.016691685" Jul 10 22:51:39.098787 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.098749 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:39.099324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.098938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:51:39.158421 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.158386 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-87cdc47f5-2nvpm"] Jul 10 22:51:39.158726 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.158666 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" podUID="30465094-d587-43de-8fe5-648e5f468ab1" containerName="manager" containerID="cri-o://de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94" gracePeriod=30 Jul 10 22:51:39.415483 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.415458 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:39.535569 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.535535 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30465094-d587-43de-8fe5-648e5f468ab1-webhook-cert\") pod \"30465094-d587-43de-8fe5-648e5f468ab1\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " Jul 10 22:51:39.535749 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.535577 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfp5s\" (UniqueName: \"kubernetes.io/projected/30465094-d587-43de-8fe5-648e5f468ab1-kube-api-access-zfp5s\") pod \"30465094-d587-43de-8fe5-648e5f468ab1\" (UID: \"30465094-d587-43de-8fe5-648e5f468ab1\") " Jul 10 22:51:39.537766 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.537740 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30465094-d587-43de-8fe5-648e5f468ab1-kube-api-access-zfp5s" (OuterVolumeSpecName: "kube-api-access-zfp5s") pod "30465094-d587-43de-8fe5-648e5f468ab1" (UID: "30465094-d587-43de-8fe5-648e5f468ab1"). InnerVolumeSpecName "kube-api-access-zfp5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:51:39.537878 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.537824 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30465094-d587-43de-8fe5-648e5f468ab1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "30465094-d587-43de-8fe5-648e5f468ab1" (UID: "30465094-d587-43de-8fe5-648e5f468ab1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:51:39.636252 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.636221 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30465094-d587-43de-8fe5-648e5f468ab1-webhook-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:51:39.636252 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:39.636246 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfp5s\" (UniqueName: \"kubernetes.io/projected/30465094-d587-43de-8fe5-648e5f468ab1-kube-api-access-zfp5s\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:51:40.150055 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.150024 2577 generic.go:356] "Generic (PLEG): container finished" podID="30465094-d587-43de-8fe5-648e5f468ab1" containerID="de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94" exitCode=0 Jul 10 22:51:40.150485 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.150073 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" Jul 10 22:51:40.150485 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.150119 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" event={"ID":"30465094-d587-43de-8fe5-648e5f468ab1","Type":"ContainerDied","Data":"de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94"} Jul 10 22:51:40.150485 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.150168 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87cdc47f5-2nvpm" event={"ID":"30465094-d587-43de-8fe5-648e5f468ab1","Type":"ContainerDied","Data":"42e575b5da8f0d67d86b6ccbc23da1515400d7205a5228f3822143e72c74ff29"} Jul 10 22:51:40.150485 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.150196 2577 scope.go:117] "RemoveContainer" containerID="de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94" Jul 10 22:51:40.158676 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.158660 2577 scope.go:117] "RemoveContainer" containerID="de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94" Jul 10 22:51:40.158927 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:51:40.158909 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94\": container with ID starting with de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94 not found: ID does not exist" containerID="de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94" Jul 10 22:51:40.159000 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.158934 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94"} err="failed to get container status \"de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94\": rpc error: code = NotFound desc = could not find container \"de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94\": container with ID starting with de80dc951ac78e63040944d820c86ae6e7289ea1323737784485dec36a860d94 not found: ID does not exist" Jul 10 22:51:40.167491 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.167466 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-87cdc47f5-2nvpm"] Jul 10 22:51:40.170668 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:40.170645 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-87cdc47f5-2nvpm"] Jul 10 22:51:41.635102 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:51:41.635069 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30465094-d587-43de-8fe5-648e5f468ab1" path="/var/lib/kubelet/pods/30465094-d587-43de-8fe5-648e5f468ab1/volumes" Jul 10 22:52:01.070092 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.070008 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4"] Jul 10 22:52:01.070447 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.070346 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30465094-d587-43de-8fe5-648e5f468ab1" containerName="manager" Jul 10 22:52:01.070447 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.070356 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="30465094-d587-43de-8fe5-648e5f468ab1" containerName="manager" Jul 10 22:52:01.070447 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.070430 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="30465094-d587-43de-8fe5-648e5f468ab1" containerName="manager" Jul 10 22:52:01.076650 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.076631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.079898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.079869 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 22:52:01.079898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.079890 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 22:52:01.080956 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.080937 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 22:52:01.081084 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.080966 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s9qhs\"" Jul 10 22:52:01.087503 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.087480 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4"] Jul 10 22:52:01.116213 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.116180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209be334-a47f-4c80-bdee-cc943854a0fa-openshift-service-ca\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.116370 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.116279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pndj\" (UniqueName: \"kubernetes.io/projected/209be334-a47f-4c80-bdee-cc943854a0fa-kube-api-access-9pndj\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.116370 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.116328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209be334-a47f-4c80-bdee-cc943854a0fa-maas-api-tls\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.217739 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.217692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pndj\" (UniqueName: \"kubernetes.io/projected/209be334-a47f-4c80-bdee-cc943854a0fa-kube-api-access-9pndj\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.217914 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.217757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209be334-a47f-4c80-bdee-cc943854a0fa-maas-api-tls\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.217914 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.217787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209be334-a47f-4c80-bdee-cc943854a0fa-openshift-service-ca\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.217914 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:52:01.217896 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 10 22:52:01.218031 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:52:01.217986 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209be334-a47f-4c80-bdee-cc943854a0fa-maas-api-tls podName:209be334-a47f-4c80-bdee-cc943854a0fa nodeName:}" failed. No retries permitted until 2026-07-10 22:52:01.717950156 +0000 UTC m=+540.595641765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/209be334-a47f-4c80-bdee-cc943854a0fa-maas-api-tls") pod "maas-api-564ccfc6f7-wg9v4" (UID: "209be334-a47f-4c80-bdee-cc943854a0fa") : secret "maas-api-serving-cert" not found Jul 10 22:52:01.218469 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.218449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209be334-a47f-4c80-bdee-cc943854a0fa-openshift-service-ca\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.227005 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.226961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pndj\" (UniqueName: \"kubernetes.io/projected/209be334-a47f-4c80-bdee-cc943854a0fa-kube-api-access-9pndj\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.722278 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.722239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209be334-a47f-4c80-bdee-cc943854a0fa-maas-api-tls\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.724739 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.724714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209be334-a47f-4c80-bdee-cc943854a0fa-maas-api-tls\") pod \"maas-api-564ccfc6f7-wg9v4\" (UID: \"209be334-a47f-4c80-bdee-cc943854a0fa\") " pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:01.991218 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.991164 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s9qhs\"" Jul 10 22:52:01.999451 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:01.999424 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:02.144129 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:02.143858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4"] Jul 10 22:52:02.148350 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:52:02.148315 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209be334_a47f_4c80_bdee_cc943854a0fa.slice/crio-2ce2b1f7b9c06fc6c90307afe8ad2ea5ebd2fab39f89fa5840a4b6d211189237 WatchSource:0}: Error finding container 2ce2b1f7b9c06fc6c90307afe8ad2ea5ebd2fab39f89fa5840a4b6d211189237: Status 404 returned error can't find the container with id 2ce2b1f7b9c06fc6c90307afe8ad2ea5ebd2fab39f89fa5840a4b6d211189237 Jul 10 22:52:02.262631 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:02.262546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" event={"ID":"209be334-a47f-4c80-bdee-cc943854a0fa","Type":"ContainerStarted","Data":"2ce2b1f7b9c06fc6c90307afe8ad2ea5ebd2fab39f89fa5840a4b6d211189237"} Jul 10 22:52:04.275297 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:04.275255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" event={"ID":"209be334-a47f-4c80-bdee-cc943854a0fa","Type":"ContainerStarted","Data":"1a7f7191c20ffcc42f60adcfcfc358aec62768206a7579eb157111a41e17bdc9"} Jul 10 22:52:04.275682 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:04.275347 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:04.293558 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:04.293512 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" podStartSLOduration=1.820658324 podStartE2EDuration="3.29349586s" totalImagesPullingTime="1.472837536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:52:04.291418081 +0000 UTC m=+543.169109717" watchObservedRunningTime="2026-07-10 22:52:04.29349586 +0000 UTC m=+543.171187482" Jul 10 22:52:10.286206 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:10.286174 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-564ccfc6f7-wg9v4" Jul 10 22:52:18.033338 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.033303 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc"] Jul 10 22:52:18.037262 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.037240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.040230 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.040206 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-crclv\"" Jul 10 22:52:18.040377 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.040206 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 22:52:18.040377 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.040207 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 22:52:18.041172 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.041156 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 22:52:18.046599 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.046578 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc"] Jul 10 22:52:18.176588 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.176557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.176588 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.176598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.176831 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.176635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.176831 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.176662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.176831 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.176686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94df47bf-7222-4b9f-9994-af2b0d8f3c03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.176831 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.176706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5r5\" (UniqueName: \"kubernetes.io/projected/94df47bf-7222-4b9f-9994-af2b0d8f3c03-kube-api-access-qk5r5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277219 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277390 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277390 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277390 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277390 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94df47bf-7222-4b9f-9994-af2b0d8f3c03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277390 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5r5\" (UniqueName: \"kubernetes.io/projected/94df47bf-7222-4b9f-9994-af2b0d8f3c03-kube-api-access-qk5r5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277670 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277709 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277659 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.277768 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.277742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.279633 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.279602 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94df47bf-7222-4b9f-9994-af2b0d8f3c03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.279821 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.279768 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94df47bf-7222-4b9f-9994-af2b0d8f3c03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.285033 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.284949 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5r5\" (UniqueName: \"kubernetes.io/projected/94df47bf-7222-4b9f-9994-af2b0d8f3c03-kube-api-access-qk5r5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc\" (UID: \"94df47bf-7222-4b9f-9994-af2b0d8f3c03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.349433 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.349400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:18.486395 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:18.486371 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc"] Jul 10 22:52:18.489116 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:52:18.489088 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94df47bf_7222_4b9f_9994_af2b0d8f3c03.slice/crio-c5969e12f69a18eb4e03e92202f9dae27aa09800dff425fe5f2a6489fce3536d WatchSource:0}: Error finding container c5969e12f69a18eb4e03e92202f9dae27aa09800dff425fe5f2a6489fce3536d: Status 404 returned error can't find the container with id c5969e12f69a18eb4e03e92202f9dae27aa09800dff425fe5f2a6489fce3536d Jul 10 22:52:19.353979 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:19.353939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" event={"ID":"94df47bf-7222-4b9f-9994-af2b0d8f3c03","Type":"ContainerStarted","Data":"c5969e12f69a18eb4e03e92202f9dae27aa09800dff425fe5f2a6489fce3536d"} Jul 10 22:52:20.360259 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:20.360226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" event={"ID":"94df47bf-7222-4b9f-9994-af2b0d8f3c03","Type":"ContainerStarted","Data":"ec911184ac55a003a207b0fe7902abe1eaf24f7eac2119a5e6581c99b0303747"} Jul 10 22:52:20.377083 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:20.377033 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" podStartSLOduration=1.273046267 podStartE2EDuration="2.37701736s" totalImagesPullingTime="1.103971093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:52:20.376759073 +0000 UTC m=+559.254450697" watchObservedRunningTime="2026-07-10 22:52:20.37701736 +0000 UTC m=+559.254708979" Jul 10 22:52:25.824279 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.824240 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7"] Jul 10 22:52:25.827757 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.827732 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:25.830682 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.830661 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 22:52:25.837767 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.837745 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7"] Jul 10 22:52:25.948661 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.948629 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7b2h\" (UniqueName: \"kubernetes.io/projected/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-kube-api-access-w7b2h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:25.948924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.948670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:25.948924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.948691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:25.948924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.948816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:25.948924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.948860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:25.948924 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:25.948916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050281 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050281 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050514 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050514 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b2h\" (UniqueName: \"kubernetes.io/projected/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-kube-api-access-w7b2h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050514 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050514 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050714 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.050798 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.051029 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.050999 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.052689 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.052658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.052785 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.052758 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.059422 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.059401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b2h\" (UniqueName: \"kubernetes.io/projected/fd049c08-4c89-4a6a-9c65-2d461cb8ead7-kube-api-access-w7b2h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c8mx7\" (UID: \"fd049c08-4c89-4a6a-9c65-2d461cb8ead7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.141356 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.141270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:26.275729 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.275693 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7"] Jul 10 22:52:26.279088 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:52:26.279061 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd049c08_4c89_4a6a_9c65_2d461cb8ead7.slice/crio-c57a4af6603c03c6074f28f874623e7295e5851c8a176cccb17598855e37bb37 WatchSource:0}: Error finding container c57a4af6603c03c6074f28f874623e7295e5851c8a176cccb17598855e37bb37: Status 404 returned error can't find the container with id c57a4af6603c03c6074f28f874623e7295e5851c8a176cccb17598855e37bb37 Jul 10 22:52:26.393819 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:26.393728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" event={"ID":"fd049c08-4c89-4a6a-9c65-2d461cb8ead7","Type":"ContainerStarted","Data":"c57a4af6603c03c6074f28f874623e7295e5851c8a176cccb17598855e37bb37"} Jul 10 22:52:27.401513 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:27.401474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" event={"ID":"fd049c08-4c89-4a6a-9c65-2d461cb8ead7","Type":"ContainerStarted","Data":"ac9d8805d1c18ac9db66190f52cfb5a1019244998af66c516d35884b2661cee5"} Jul 10 22:52:27.420106 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:27.420055 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" podStartSLOduration=2.241052534 podStartE2EDuration="2.420038795s" totalImagesPullingTime="178.986261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:52:27.417634209 +0000 UTC m=+566.295325843" watchObservedRunningTime="2026-07-10 22:52:27.420038795 +0000 UTC m=+566.297730471" Jul 10 22:52:28.350281 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:28.350247 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:28.350496 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:28.350314 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:28.366306 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:28.366277 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:28.418306 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:28.418270 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc" Jul 10 22:52:36.142242 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:36.142194 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:36.142666 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:36.142273 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:36.154951 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:36.154919 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:52:36.464806 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:52:36.464774 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c8mx7" Jul 10 22:53:01.531506 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:53:01.531480 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:53:01.532057 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:53:01.532005 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:55:15.863674 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:15.863596 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-547499fdd5-hwfj6"] Jul 10 22:55:15.864157 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:15.863793 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-547499fdd5-hwfj6" podUID="4fc0d071-494b-47a3-ac10-ba521af363d9" containerName="manager" containerID="cri-o://1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7" gracePeriod=30 Jul 10 22:55:16.115393 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.115338 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:55:16.133555 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.133532 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fc0d071-494b-47a3-ac10-ba521af363d9-webhook-cert\") pod \"4fc0d071-494b-47a3-ac10-ba521af363d9\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " Jul 10 22:55:16.133695 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.133603 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6vff\" (UniqueName: \"kubernetes.io/projected/4fc0d071-494b-47a3-ac10-ba521af363d9-kube-api-access-m6vff\") pod \"4fc0d071-494b-47a3-ac10-ba521af363d9\" (UID: \"4fc0d071-494b-47a3-ac10-ba521af363d9\") " Jul 10 22:55:16.135522 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.135491 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc0d071-494b-47a3-ac10-ba521af363d9-kube-api-access-m6vff" (OuterVolumeSpecName: "kube-api-access-m6vff") pod "4fc0d071-494b-47a3-ac10-ba521af363d9" (UID: "4fc0d071-494b-47a3-ac10-ba521af363d9"). InnerVolumeSpecName "kube-api-access-m6vff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:55:16.135752 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.135532 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc0d071-494b-47a3-ac10-ba521af363d9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4fc0d071-494b-47a3-ac10-ba521af363d9" (UID: "4fc0d071-494b-47a3-ac10-ba521af363d9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:55:16.235081 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.235052 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fc0d071-494b-47a3-ac10-ba521af363d9-webhook-cert\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:55:16.235081 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.235080 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6vff\" (UniqueName: \"kubernetes.io/projected/4fc0d071-494b-47a3-ac10-ba521af363d9-kube-api-access-m6vff\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:55:16.285033 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.285004 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fc0d071-494b-47a3-ac10-ba521af363d9" containerID="1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7" exitCode=0 Jul 10 22:55:16.285202 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.285104 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-547499fdd5-hwfj6" Jul 10 22:55:16.285202 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.285142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-547499fdd5-hwfj6" event={"ID":"4fc0d071-494b-47a3-ac10-ba521af363d9","Type":"ContainerDied","Data":"1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7"} Jul 10 22:55:16.285324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.285202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-547499fdd5-hwfj6" event={"ID":"4fc0d071-494b-47a3-ac10-ba521af363d9","Type":"ContainerDied","Data":"3a06ca9901dc939a988549e2d3730670dc9490a7c7f2e4e7f55a0332a2cc477d"} Jul 10 22:55:16.285324 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.285232 2577 scope.go:117] "RemoveContainer" containerID="1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7" Jul 10 22:55:16.294260 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.294244 2577 scope.go:117] "RemoveContainer" containerID="1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7" Jul 10 22:55:16.294481 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:55:16.294461 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7\": container with ID starting with 1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7 not found: ID does not exist" containerID="1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7" Jul 10 22:55:16.294538 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.294487 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7"} err="failed to get container status \"1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7\": rpc error: code = NotFound desc = could not find container \"1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7\": container with ID starting with 1263e9a566e58e5e4faf3345785726472ce0c66cf8c6c2f2a598fb2601bb10f7 not found: ID does not exist" Jul 10 22:55:16.309323 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.309293 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-547499fdd5-hwfj6"] Jul 10 22:55:16.312603 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:16.312582 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-547499fdd5-hwfj6"] Jul 10 22:55:17.081406 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.081365 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-547499fdd5-8bvqd"] Jul 10 22:55:17.081927 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.081908 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc0d071-494b-47a3-ac10-ba521af363d9" containerName="manager" Jul 10 22:55:17.082092 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.081928 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc0d071-494b-47a3-ac10-ba521af363d9" containerName="manager" Jul 10 22:55:17.082092 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.082041 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc0d071-494b-47a3-ac10-ba521af363d9" containerName="manager" Jul 10 22:55:17.086361 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.086340 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.089336 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.089319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wj8k9\"" Jul 10 22:55:17.090451 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.090432 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 22:55:17.090564 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.090453 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 22:55:17.095956 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.095928 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-547499fdd5-8bvqd"] Jul 10 22:55:17.142304 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.142274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42800ff6-a893-44e8-8064-7cb3cf69267e-webhook-cert\") pod \"maas-controller-547499fdd5-8bvqd\" (UID: \"42800ff6-a893-44e8-8064-7cb3cf69267e\") " pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.142420 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.142333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsf5\" (UniqueName: \"kubernetes.io/projected/42800ff6-a893-44e8-8064-7cb3cf69267e-kube-api-access-bcsf5\") pod \"maas-controller-547499fdd5-8bvqd\" (UID: \"42800ff6-a893-44e8-8064-7cb3cf69267e\") " pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.243546 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.243521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42800ff6-a893-44e8-8064-7cb3cf69267e-webhook-cert\") pod \"maas-controller-547499fdd5-8bvqd\" (UID: \"42800ff6-a893-44e8-8064-7cb3cf69267e\") " pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.243651 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.243563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsf5\" (UniqueName: \"kubernetes.io/projected/42800ff6-a893-44e8-8064-7cb3cf69267e-kube-api-access-bcsf5\") pod \"maas-controller-547499fdd5-8bvqd\" (UID: \"42800ff6-a893-44e8-8064-7cb3cf69267e\") " pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.245898 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.245878 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42800ff6-a893-44e8-8064-7cb3cf69267e-webhook-cert\") pod \"maas-controller-547499fdd5-8bvqd\" (UID: \"42800ff6-a893-44e8-8064-7cb3cf69267e\") " pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.253095 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.253076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsf5\" (UniqueName: \"kubernetes.io/projected/42800ff6-a893-44e8-8064-7cb3cf69267e-kube-api-access-bcsf5\") pod \"maas-controller-547499fdd5-8bvqd\" (UID: \"42800ff6-a893-44e8-8064-7cb3cf69267e\") " pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.397341 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.397266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:17.525197 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.525171 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-547499fdd5-8bvqd"] Jul 10 22:55:17.527138 ip-10-0-139-196 kubenswrapper[2577]: W0710 22:55:17.527114 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42800ff6_a893_44e8_8064_7cb3cf69267e.slice/crio-7d95b714654a9af911d39c39b6039ad21f714ba61c87866bf99d413b67418cf4 WatchSource:0}: Error finding container 7d95b714654a9af911d39c39b6039ad21f714ba61c87866bf99d413b67418cf4: Status 404 returned error can't find the container with id 7d95b714654a9af911d39c39b6039ad21f714ba61c87866bf99d413b67418cf4 Jul 10 22:55:17.528447 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.528430 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:55:17.635526 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:17.635499 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc0d071-494b-47a3-ac10-ba521af363d9" path="/var/lib/kubelet/pods/4fc0d071-494b-47a3-ac10-ba521af363d9/volumes" Jul 10 22:55:18.298209 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:18.298116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-547499fdd5-8bvqd" event={"ID":"42800ff6-a893-44e8-8064-7cb3cf69267e","Type":"ContainerStarted","Data":"ed65a76083ec1ead8859de20e2ec388e0c4d8076f5f81e80b9987aee5d002e3a"} Jul 10 22:55:18.298209 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:18.298181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-547499fdd5-8bvqd" event={"ID":"42800ff6-a893-44e8-8064-7cb3cf69267e","Type":"ContainerStarted","Data":"7d95b714654a9af911d39c39b6039ad21f714ba61c87866bf99d413b67418cf4"} Jul 10 22:55:18.298209 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:18.298206 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:55:18.322938 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:18.322878 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-547499fdd5-8bvqd" podStartSLOduration=0.832287026 podStartE2EDuration="1.322861605s" totalImagesPullingTime="490.574579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:55:18.321588382 +0000 UTC m=+737.199280023" watchObservedRunningTime="2026-07-10 22:55:18.322861605 +0000 UTC m=+737.200553223" Jul 10 22:55:29.328417 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:55:29.328379 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-547499fdd5-8bvqd" Jul 10 22:58:01.566302 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:58:01.566227 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:58:01.567561 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:58:01.567539 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 22:59:50.992560 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:50.992527 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6"] Jul 10 22:59:50.994966 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:50.992714 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" podUID="53885095-5123-468e-b4f0-e94aea3c2189" containerName="manager" containerID="cri-o://006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456" gracePeriod=10 Jul 10 22:59:54.655298 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.655275 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:59:54.701063 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.700964 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53885095-5123-468e-b4f0-e94aea3c2189-extensions-socket-volume\") pod \"53885095-5123-468e-b4f0-e94aea3c2189\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " Jul 10 22:59:54.701212 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.701065 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt75d\" (UniqueName: \"kubernetes.io/projected/53885095-5123-468e-b4f0-e94aea3c2189-kube-api-access-pt75d\") pod \"53885095-5123-468e-b4f0-e94aea3c2189\" (UID: \"53885095-5123-468e-b4f0-e94aea3c2189\") " Jul 10 22:59:54.701302 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.701280 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53885095-5123-468e-b4f0-e94aea3c2189-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "53885095-5123-468e-b4f0-e94aea3c2189" (UID: "53885095-5123-468e-b4f0-e94aea3c2189"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:59:54.701378 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.701365 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53885095-5123-468e-b4f0-e94aea3c2189-extensions-socket-volume\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:59:54.703201 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.703169 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53885095-5123-468e-b4f0-e94aea3c2189-kube-api-access-pt75d" (OuterVolumeSpecName: "kube-api-access-pt75d") pod "53885095-5123-468e-b4f0-e94aea3c2189" (UID: "53885095-5123-468e-b4f0-e94aea3c2189"). InnerVolumeSpecName "kube-api-access-pt75d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:59:54.743065 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.743031 2577 generic.go:356] "Generic (PLEG): container finished" podID="53885095-5123-468e-b4f0-e94aea3c2189" containerID="006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456" exitCode=0 Jul 10 22:59:54.743192 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.743072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" event={"ID":"53885095-5123-468e-b4f0-e94aea3c2189","Type":"ContainerDied","Data":"006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456"} Jul 10 22:59:54.743192 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.743091 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" Jul 10 22:59:54.743192 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.743146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6" event={"ID":"53885095-5123-468e-b4f0-e94aea3c2189","Type":"ContainerDied","Data":"a8dff176a7112cce10b93d58276d74aa6d4367bd23fcfc7625617688e30cb803"} Jul 10 22:59:54.743192 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.743169 2577 scope.go:117] "RemoveContainer" containerID="006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456" Jul 10 22:59:54.753178 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.753164 2577 scope.go:117] "RemoveContainer" containerID="006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456" Jul 10 22:59:54.753411 ip-10-0-139-196 kubenswrapper[2577]: E0710 22:59:54.753391 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456\": container with ID starting with 006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456 not found: ID does not exist" containerID="006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456" Jul 10 22:59:54.753457 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.753418 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456"} err="failed to get container status \"006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456\": rpc error: code = NotFound desc = could not find container \"006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456\": container with ID starting with 006a7c36346a356cb15e00ed411c90fb2280633c059d85b05fd00f58ccfdc456 not found: ID does not exist" Jul 10 22:59:54.764874 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.764848 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6"] Jul 10 22:59:54.768525 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.768505 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85qc6"] Jul 10 22:59:54.802707 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:54.802681 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt75d\" (UniqueName: \"kubernetes.io/projected/53885095-5123-468e-b4f0-e94aea3c2189-kube-api-access-pt75d\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 22:59:55.636175 ip-10-0-139-196 kubenswrapper[2577]: I0710 22:59:55.636142 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53885095-5123-468e-b4f0-e94aea3c2189" path="/var/lib/kubelet/pods/53885095-5123-468e-b4f0-e94aea3c2189/volumes" Jul 10 23:00:00.156648 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.156615 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62"] Jul 10 23:00:00.157041 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.156965 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53885095-5123-468e-b4f0-e94aea3c2189" containerName="manager" Jul 10 23:00:00.157041 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.156989 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53885095-5123-468e-b4f0-e94aea3c2189" containerName="manager" Jul 10 23:00:00.157138 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.157065 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="53885095-5123-468e-b4f0-e94aea3c2189" containerName="manager" Jul 10 23:00:00.159541 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.159526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" Jul 10 23:00:00.164302 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.164283 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dh5r5\"" Jul 10 23:00:00.169197 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.169176 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62"] Jul 10 23:00:00.169589 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.169575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" Jul 10 23:00:00.313549 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.313529 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62"] Jul 10 23:00:00.315705 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:00:00.315676 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd847e569_0eea_4d52_9d29_5f2a0d8962ef.slice/crio-dd990dc2a2e0a7be86abff4ad79fd7db1dc45747a3e3129e1085452f535cc2e6 WatchSource:0}: Error finding container dd990dc2a2e0a7be86abff4ad79fd7db1dc45747a3e3129e1085452f535cc2e6: Status 404 returned error can't find the container with id dd990dc2a2e0a7be86abff4ad79fd7db1dc45747a3e3129e1085452f535cc2e6 Jul 10 23:00:00.774484 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:00.774449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" event={"ID":"d847e569-0eea-4d52-9d29-5f2a0d8962ef","Type":"ContainerStarted","Data":"dd990dc2a2e0a7be86abff4ad79fd7db1dc45747a3e3129e1085452f535cc2e6"} Jul 10 23:00:05.804764 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:05.804732 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" event={"ID":"d847e569-0eea-4d52-9d29-5f2a0d8962ef","Type":"ContainerStarted","Data":"78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d"} Jul 10 23:00:05.822593 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:05.822546 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" podStartSLOduration=1.029341446 podStartE2EDuration="5.82253127s" totalImagesPullingTime="4.793189824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:00:05.820689579 +0000 UTC m=+1024.698381222" watchObservedRunningTime="2026-07-10 23:00:05.82253127 +0000 UTC m=+1024.700222890" Jul 10 23:00:52.102953 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.102917 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5"] Jul 10 23:00:52.106194 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.106177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.109235 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.109214 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tj8sl\"" Jul 10 23:00:52.120962 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.120936 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5"] Jul 10 23:00:52.179983 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.179934 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17d4265-b6ec-44b5-b12b-c2018c2ffc2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-48lm5\" (UID: \"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.180141 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.180044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbmqv\" (UniqueName: \"kubernetes.io/projected/e17d4265-b6ec-44b5-b12b-c2018c2ffc2a-kube-api-access-zbmqv\") pod \"kuadrant-operator-controller-manager-86878c769f-48lm5\" (UID: \"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.280509 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.280459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmqv\" (UniqueName: \"kubernetes.io/projected/e17d4265-b6ec-44b5-b12b-c2018c2ffc2a-kube-api-access-zbmqv\") pod \"kuadrant-operator-controller-manager-86878c769f-48lm5\" (UID: \"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.280657 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.280536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17d4265-b6ec-44b5-b12b-c2018c2ffc2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-48lm5\" (UID: \"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.280915 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.280896 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e17d4265-b6ec-44b5-b12b-c2018c2ffc2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-48lm5\" (UID: \"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.296454 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.296433 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmqv\" (UniqueName: \"kubernetes.io/projected/e17d4265-b6ec-44b5-b12b-c2018c2ffc2a-kube-api-access-zbmqv\") pod \"kuadrant-operator-controller-manager-86878c769f-48lm5\" (UID: \"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.416238 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.416199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:52.549985 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:52.549951 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5"] Jul 10 23:00:52.552304 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:00:52.552266 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17d4265_b6ec_44b5_b12b_c2018c2ffc2a.slice/crio-eb59c08e88a1f6d58dd57e0e82f3ae1e51c8a7c1487b6defb48c2516b51cde9d WatchSource:0}: Error finding container eb59c08e88a1f6d58dd57e0e82f3ae1e51c8a7c1487b6defb48c2516b51cde9d: Status 404 returned error can't find the container with id eb59c08e88a1f6d58dd57e0e82f3ae1e51c8a7c1487b6defb48c2516b51cde9d Jul 10 23:00:53.044034 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:53.044003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" event={"ID":"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a","Type":"ContainerStarted","Data":"30dafe439ed31bb47207e8f291efa5313cf12079384b747ddc99064646d4ead8"} Jul 10 23:00:53.044218 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:53.044069 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:00:53.044218 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:53.044093 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" event={"ID":"e17d4265-b6ec-44b5-b12b-c2018c2ffc2a","Type":"ContainerStarted","Data":"eb59c08e88a1f6d58dd57e0e82f3ae1e51c8a7c1487b6defb48c2516b51cde9d"} Jul 10 23:00:53.068539 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:00:53.068474 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" podStartSLOduration=1.068458733 podStartE2EDuration="1.068458733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:00:53.066619287 +0000 UTC m=+1071.944310932" watchObservedRunningTime="2026-07-10 23:00:53.068458733 +0000 UTC m=+1071.946150353" Jul 10 23:01:00.011400 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:00.011365 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62"] Jul 10 23:01:00.011879 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:00.011586 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" podUID="d847e569-0eea-4d52-9d29-5f2a0d8962ef" containerName="cleanup" containerID="cri-o://78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d" gracePeriod=30 Jul 10 23:01:04.051917 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:04.051886 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-48lm5" Jul 10 23:01:30.206192 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.206163 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" Jul 10 23:01:30.235893 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.235857 2577 generic.go:356] "Generic (PLEG): container finished" podID="d847e569-0eea-4d52-9d29-5f2a0d8962ef" containerID="78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d" exitCode=137 Jul 10 23:01:30.236083 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.235964 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" Jul 10 23:01:30.236083 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.235984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" event={"ID":"d847e569-0eea-4d52-9d29-5f2a0d8962ef","Type":"ContainerDied","Data":"78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d"} Jul 10 23:01:30.236198 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.236122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62" event={"ID":"d847e569-0eea-4d52-9d29-5f2a0d8962ef","Type":"ContainerDied","Data":"dd990dc2a2e0a7be86abff4ad79fd7db1dc45747a3e3129e1085452f535cc2e6"} Jul 10 23:01:30.236198 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.236161 2577 scope.go:117] "RemoveContainer" containerID="78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d" Jul 10 23:01:30.246614 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.246594 2577 scope.go:117] "RemoveContainer" containerID="78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d" Jul 10 23:01:30.246869 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:01:30.246850 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d\": container with ID starting with 78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d not found: ID does not exist" containerID="78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d" Jul 10 23:01:30.246938 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.246881 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d"} err="failed to get container status \"78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d\": rpc error: code = NotFound desc = could not find container \"78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d\": container with ID starting with 78d86ea20e2d251dc1fbc0b6ffc1439f7c4d714f55f2efc33d6dca1ef1e4666d not found: ID does not exist" Jul 10 23:01:30.274217 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.274158 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62"] Jul 10 23:01:30.277297 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:30.277277 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728740-9rp62"] Jul 10 23:01:31.641623 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:01:31.641587 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d847e569-0eea-4d52-9d29-5f2a0d8962ef" path="/var/lib/kubelet/pods/d847e569-0eea-4d52-9d29-5f2a0d8962ef/volumes" Jul 10 23:03:01.604347 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:03:01.604313 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:03:01.604894 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:03:01.604564 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:07:11.704626 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.704597 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g"] Jul 10 23:07:11.705058 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.704945 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d847e569-0eea-4d52-9d29-5f2a0d8962ef" containerName="cleanup" Jul 10 23:07:11.705058 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.704955 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d847e569-0eea-4d52-9d29-5f2a0d8962ef" containerName="cleanup" Jul 10 23:07:11.705058 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.705033 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d847e569-0eea-4d52-9d29-5f2a0d8962ef" containerName="cleanup" Jul 10 23:07:11.707821 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.707806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.710390 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.710374 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8361fb0e-serving-cert\"" Jul 10 23:07:11.720931 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.720906 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g"] Jul 10 23:07:11.773222 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.773196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.773353 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.773232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmkg\" (UniqueName: \"kubernetes.io/projected/bceb4d03-35bd-4510-be2d-6db863ce14d7-kube-api-access-qjmkg\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.773353 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.773313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceb4d03-35bd-4510-be2d-6db863ce14d7-openshift-service-ca\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.873759 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.873733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.873914 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.873766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmkg\" (UniqueName: \"kubernetes.io/projected/bceb4d03-35bd-4510-be2d-6db863ce14d7-kube-api-access-qjmkg\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.873914 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.873812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceb4d03-35bd-4510-be2d-6db863ce14d7-openshift-service-ca\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.873914 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:11.873888 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-serving-cert: secret "maas-api-e2e-ait-8361fb0e-serving-cert" not found Jul 10 23:07:11.874043 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:11.873962 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls podName:bceb4d03-35bd-4510-be2d-6db863ce14d7 nodeName:}" failed. No retries permitted until 2026-07-10 23:07:12.373942886 +0000 UTC m=+1451.251634504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls") pod "maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" (UID: "bceb4d03-35bd-4510-be2d-6db863ce14d7") : secret "maas-api-e2e-ait-8361fb0e-serving-cert" not found Jul 10 23:07:11.874386 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.874370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceb4d03-35bd-4510-be2d-6db863ce14d7-openshift-service-ca\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:11.882966 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:11.882938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmkg\" (UniqueName: \"kubernetes.io/projected/bceb4d03-35bd-4510-be2d-6db863ce14d7-kube-api-access-qjmkg\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:12.377577 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:12.377546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:12.379852 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:12.379830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls\") pod \"maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:12.618333 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:12.618299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:12.772746 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:12.772700 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g"] Jul 10 23:07:12.774896 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:07:12.774859 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbceb4d03_35bd_4510_be2d_6db863ce14d7.slice/crio-d4d6ff5549a6102dc8b5250395fbfd3eb366fdeeb46c5a1b5643ea8e8492f429 WatchSource:0}: Error finding container d4d6ff5549a6102dc8b5250395fbfd3eb366fdeeb46c5a1b5643ea8e8492f429: Status 404 returned error can't find the container with id d4d6ff5549a6102dc8b5250395fbfd3eb366fdeeb46c5a1b5643ea8e8492f429 Jul 10 23:07:12.776276 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:12.776257 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 23:07:13.020477 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:13.020442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" event={"ID":"bceb4d03-35bd-4510-be2d-6db863ce14d7","Type":"ContainerStarted","Data":"d4d6ff5549a6102dc8b5250395fbfd3eb366fdeeb46c5a1b5643ea8e8492f429"} Jul 10 23:07:14.026646 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:14.026613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" event={"ID":"bceb4d03-35bd-4510-be2d-6db863ce14d7","Type":"ContainerStarted","Data":"11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2"} Jul 10 23:07:14.027045 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:14.026680 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:14.045060 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:14.045007 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" podStartSLOduration=2.504783654 podStartE2EDuration="3.044992126s" totalImagesPullingTime="540.208472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:07:14.042820462 +0000 UTC m=+1452.920512083" watchObservedRunningTime="2026-07-10 23:07:14.044992126 +0000 UTC m=+1452.922683742" Jul 10 23:07:17.167761 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.167723 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g"] Jul 10 23:07:17.168165 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.167999 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" podUID="bceb4d03-35bd-4510-be2d-6db863ce14d7" containerName="maas-api" containerID="cri-o://11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2" gracePeriod=30 Jul 10 23:07:17.411270 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.411246 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:17.522650 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.522587 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjmkg\" (UniqueName: \"kubernetes.io/projected/bceb4d03-35bd-4510-be2d-6db863ce14d7-kube-api-access-qjmkg\") pod \"bceb4d03-35bd-4510-be2d-6db863ce14d7\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " Jul 10 23:07:17.522650 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.522643 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls\") pod \"bceb4d03-35bd-4510-be2d-6db863ce14d7\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " Jul 10 23:07:17.522802 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.522692 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceb4d03-35bd-4510-be2d-6db863ce14d7-openshift-service-ca\") pod \"bceb4d03-35bd-4510-be2d-6db863ce14d7\" (UID: \"bceb4d03-35bd-4510-be2d-6db863ce14d7\") " Jul 10 23:07:17.523076 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.523052 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bceb4d03-35bd-4510-be2d-6db863ce14d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bceb4d03-35bd-4510-be2d-6db863ce14d7" (UID: "bceb4d03-35bd-4510-be2d-6db863ce14d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:07:17.524682 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.524664 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bceb4d03-35bd-4510-be2d-6db863ce14d7-kube-api-access-qjmkg" (OuterVolumeSpecName: "kube-api-access-qjmkg") pod "bceb4d03-35bd-4510-be2d-6db863ce14d7" (UID: "bceb4d03-35bd-4510-be2d-6db863ce14d7"). InnerVolumeSpecName "kube-api-access-qjmkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:07:17.524682 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.524676 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bceb4d03-35bd-4510-be2d-6db863ce14d7" (UID: "bceb4d03-35bd-4510-be2d-6db863ce14d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:07:17.623542 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.623516 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bceb4d03-35bd-4510-be2d-6db863ce14d7-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:17.623542 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.623540 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bceb4d03-35bd-4510-be2d-6db863ce14d7-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:17.623707 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:17.623550 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjmkg\" (UniqueName: \"kubernetes.io/projected/bceb4d03-35bd-4510-be2d-6db863ce14d7-kube-api-access-qjmkg\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:18.050171 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.050137 2577 generic.go:356] "Generic (PLEG): container finished" podID="bceb4d03-35bd-4510-be2d-6db863ce14d7" containerID="11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2" exitCode=0 Jul 10 23:07:18.050333 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.050183 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" Jul 10 23:07:18.050333 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.050222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" event={"ID":"bceb4d03-35bd-4510-be2d-6db863ce14d7","Type":"ContainerDied","Data":"11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2"} Jul 10 23:07:18.050333 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.050276 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g" event={"ID":"bceb4d03-35bd-4510-be2d-6db863ce14d7","Type":"ContainerDied","Data":"d4d6ff5549a6102dc8b5250395fbfd3eb366fdeeb46c5a1b5643ea8e8492f429"} Jul 10 23:07:18.050333 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.050300 2577 scope.go:117] "RemoveContainer" containerID="11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2" Jul 10 23:07:18.058841 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.058824 2577 scope.go:117] "RemoveContainer" containerID="11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2" Jul 10 23:07:18.059154 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:18.059128 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2\": container with ID starting with 11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2 not found: ID does not exist" containerID="11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2" Jul 10 23:07:18.059212 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.059162 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2"} err="failed to get container status \"11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2\": rpc error: code = NotFound desc = could not find container \"11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2\": container with ID starting with 11aa83f1384459fe33f637e105a5cfa84de9a494f11bf196e5e42181390a56a2 not found: ID does not exist" Jul 10 23:07:18.087188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.087166 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g"] Jul 10 23:07:18.092750 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:18.092732 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8361fb0e-c6d57f99c-bxv8g"] Jul 10 23:07:19.635933 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:19.635891 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bceb4d03-35bd-4510-be2d-6db863ce14d7" path="/var/lib/kubelet/pods/bceb4d03-35bd-4510-be2d-6db863ce14d7/volumes" Jul 10 23:07:24.130584 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.130555 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp"] Jul 10 23:07:24.131087 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.131070 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bceb4d03-35bd-4510-be2d-6db863ce14d7" containerName="maas-api" Jul 10 23:07:24.131171 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.131090 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bceb4d03-35bd-4510-be2d-6db863ce14d7" containerName="maas-api" Jul 10 23:07:24.131238 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.131226 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bceb4d03-35bd-4510-be2d-6db863ce14d7" containerName="maas-api" Jul 10 23:07:24.135131 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.135110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.138356 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.138334 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-86f71747-serving-cert\"" Jul 10 23:07:24.146420 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.146393 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp"] Jul 10 23:07:24.286176 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.286144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4025fd5f-e13c-4844-9398-3e811cf37c01-openshift-service-ca\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.286176 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.286177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.286378 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.286207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwv6p\" (UniqueName: \"kubernetes.io/projected/4025fd5f-e13c-4844-9398-3e811cf37c01-kube-api-access-zwv6p\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.386730 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.386651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4025fd5f-e13c-4844-9398-3e811cf37c01-openshift-service-ca\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.386730 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.386685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.386730 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.386715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwv6p\" (UniqueName: \"kubernetes.io/projected/4025fd5f-e13c-4844-9398-3e811cf37c01-kube-api-access-zwv6p\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.387016 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:24.386819 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-serving-cert: secret "maas-api-e2e-ait-86f71747-serving-cert" not found Jul 10 23:07:24.387016 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:24.386899 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls podName:4025fd5f-e13c-4844-9398-3e811cf37c01 nodeName:}" failed. No retries permitted until 2026-07-10 23:07:24.886878621 +0000 UTC m=+1463.764570224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls") pod "maas-api-e2e-ait-86f71747-8658d99db-zkjhp" (UID: "4025fd5f-e13c-4844-9398-3e811cf37c01") : secret "maas-api-e2e-ait-86f71747-serving-cert" not found Jul 10 23:07:24.387387 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.387365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4025fd5f-e13c-4844-9398-3e811cf37c01-openshift-service-ca\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.396639 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.396613 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwv6p\" (UniqueName: \"kubernetes.io/projected/4025fd5f-e13c-4844-9398-3e811cf37c01-kube-api-access-zwv6p\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.893194 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.893166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:24.895565 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:24.895547 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls\") pod \"maas-api-e2e-ait-86f71747-8658d99db-zkjhp\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:25.047389 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:25.047350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:25.182895 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:25.182870 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp"] Jul 10 23:07:25.185593 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:07:25.185562 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4025fd5f_e13c_4844_9398_3e811cf37c01.slice/crio-04150d93f239b2cf06fe0527032b7be5b20f35db3bab9fec3fcf6b28cce5df95 WatchSource:0}: Error finding container 04150d93f239b2cf06fe0527032b7be5b20f35db3bab9fec3fcf6b28cce5df95: Status 404 returned error can't find the container with id 04150d93f239b2cf06fe0527032b7be5b20f35db3bab9fec3fcf6b28cce5df95 Jul 10 23:07:26.095445 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:26.095405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" event={"ID":"4025fd5f-e13c-4844-9398-3e811cf37c01","Type":"ContainerStarted","Data":"08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7"} Jul 10 23:07:26.095777 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:26.095475 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:26.095777 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:26.095499 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" event={"ID":"4025fd5f-e13c-4844-9398-3e811cf37c01","Type":"ContainerStarted","Data":"04150d93f239b2cf06fe0527032b7be5b20f35db3bab9fec3fcf6b28cce5df95"} Jul 10 23:07:26.117097 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:26.117048 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" podStartSLOduration=1.7551956930000001 podStartE2EDuration="2.117032589s" totalImagesPullingTime="361.836896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:07:26.115632605 +0000 UTC m=+1464.993324225" watchObservedRunningTime="2026-07-10 23:07:26.117032589 +0000 UTC m=+1464.994724208" Jul 10 23:07:29.516585 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.516555 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp"] Jul 10 23:07:29.517023 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.516733 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" podUID="4025fd5f-e13c-4844-9398-3e811cf37c01" containerName="maas-api" containerID="cri-o://08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7" gracePeriod=30 Jul 10 23:07:29.766037 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.766012 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:29.934572 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.934544 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4025fd5f-e13c-4844-9398-3e811cf37c01-openshift-service-ca\") pod \"4025fd5f-e13c-4844-9398-3e811cf37c01\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " Jul 10 23:07:29.934724 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.934607 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwv6p\" (UniqueName: \"kubernetes.io/projected/4025fd5f-e13c-4844-9398-3e811cf37c01-kube-api-access-zwv6p\") pod \"4025fd5f-e13c-4844-9398-3e811cf37c01\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " Jul 10 23:07:29.934724 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.934632 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls\") pod \"4025fd5f-e13c-4844-9398-3e811cf37c01\" (UID: \"4025fd5f-e13c-4844-9398-3e811cf37c01\") " Jul 10 23:07:29.934904 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.934882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4025fd5f-e13c-4844-9398-3e811cf37c01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4025fd5f-e13c-4844-9398-3e811cf37c01" (UID: "4025fd5f-e13c-4844-9398-3e811cf37c01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:07:29.936808 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.936785 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4025fd5f-e13c-4844-9398-3e811cf37c01" (UID: "4025fd5f-e13c-4844-9398-3e811cf37c01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:07:29.937004 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:29.936955 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4025fd5f-e13c-4844-9398-3e811cf37c01-kube-api-access-zwv6p" (OuterVolumeSpecName: "kube-api-access-zwv6p") pod "4025fd5f-e13c-4844-9398-3e811cf37c01" (UID: "4025fd5f-e13c-4844-9398-3e811cf37c01"). InnerVolumeSpecName "kube-api-access-zwv6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:07:30.036346 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.036316 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwv6p\" (UniqueName: \"kubernetes.io/projected/4025fd5f-e13c-4844-9398-3e811cf37c01-kube-api-access-zwv6p\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:30.036346 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.036340 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4025fd5f-e13c-4844-9398-3e811cf37c01-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:30.036346 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.036351 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4025fd5f-e13c-4844-9398-3e811cf37c01-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:30.118100 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.118064 2577 generic.go:356] "Generic (PLEG): container finished" podID="4025fd5f-e13c-4844-9398-3e811cf37c01" containerID="08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7" exitCode=0 Jul 10 23:07:30.118273 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.118111 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" Jul 10 23:07:30.118273 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.118147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" event={"ID":"4025fd5f-e13c-4844-9398-3e811cf37c01","Type":"ContainerDied","Data":"08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7"} Jul 10 23:07:30.118273 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.118200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp" event={"ID":"4025fd5f-e13c-4844-9398-3e811cf37c01","Type":"ContainerDied","Data":"04150d93f239b2cf06fe0527032b7be5b20f35db3bab9fec3fcf6b28cce5df95"} Jul 10 23:07:30.118273 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.118223 2577 scope.go:117] "RemoveContainer" containerID="08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7" Jul 10 23:07:30.127694 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.127677 2577 scope.go:117] "RemoveContainer" containerID="08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7" Jul 10 23:07:30.127937 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:30.127919 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7\": container with ID starting with 08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7 not found: ID does not exist" containerID="08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7" Jul 10 23:07:30.128018 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.127946 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7"} err="failed to get container status \"08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7\": rpc error: code = NotFound desc = could not find container \"08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7\": container with ID starting with 08355598f25fd5b3c8c2671cf2c73fbf8938554474f4debc239d2dfef7a4fed7 not found: ID does not exist" Jul 10 23:07:30.140503 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.140480 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp"] Jul 10 23:07:30.145416 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:30.145397 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86f71747-8658d99db-zkjhp"] Jul 10 23:07:31.641286 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:31.641246 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4025fd5f-e13c-4844-9398-3e811cf37c01" path="/var/lib/kubelet/pods/4025fd5f-e13c-4844-9398-3e811cf37c01/volumes" Jul 10 23:07:42.677724 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.677692 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg"] Jul 10 23:07:42.678233 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.678116 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4025fd5f-e13c-4844-9398-3e811cf37c01" containerName="maas-api" Jul 10 23:07:42.678233 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.678128 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4025fd5f-e13c-4844-9398-3e811cf37c01" containerName="maas-api" Jul 10 23:07:42.678233 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.678183 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4025fd5f-e13c-4844-9398-3e811cf37c01" containerName="maas-api" Jul 10 23:07:42.681259 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.681242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.684389 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.684370 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b379716f-serving-cert\"" Jul 10 23:07:42.713234 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.713213 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg"] Jul 10 23:07:42.745620 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.745585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.745770 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.745720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-openshift-service-ca\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.745770 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.745754 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8w5s\" (UniqueName: \"kubernetes.io/projected/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-kube-api-access-k8w5s\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.846910 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.846875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.847090 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.846917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-openshift-service-ca\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.847090 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.846940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8w5s\" (UniqueName: \"kubernetes.io/projected/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-kube-api-access-k8w5s\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.847090 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:42.847048 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-serving-cert: secret "maas-api-e2e-derive-b379716f-serving-cert" not found Jul 10 23:07:42.847359 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:42.847120 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls podName:a0e081a4-0d4f-4e40-bda4-652d5e4e6f44 nodeName:}" failed. No retries permitted until 2026-07-10 23:07:43.347098821 +0000 UTC m=+1482.224790433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls") pod "maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" (UID: "a0e081a4-0d4f-4e40-bda4-652d5e4e6f44") : secret "maas-api-e2e-derive-b379716f-serving-cert" not found Jul 10 23:07:42.847821 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.847795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-openshift-service-ca\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:42.860147 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:42.860122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8w5s\" (UniqueName: \"kubernetes.io/projected/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-kube-api-access-k8w5s\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:43.352249 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:43.352215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:43.354650 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:43.354617 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls\") pod \"maas-api-e2e-derive-b379716f-5b774fc776-fbgqg\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:43.591109 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:43.591073 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:43.729817 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:43.729766 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg"] Jul 10 23:07:43.731529 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:07:43.731497 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e081a4_0d4f_4e40_bda4_652d5e4e6f44.slice/crio-2b2f65b766bd4c62baf6aef08cc7df84852ae2b41304ad0a44e0639ff899f19d WatchSource:0}: Error finding container 2b2f65b766bd4c62baf6aef08cc7df84852ae2b41304ad0a44e0639ff899f19d: Status 404 returned error can't find the container with id 2b2f65b766bd4c62baf6aef08cc7df84852ae2b41304ad0a44e0639ff899f19d Jul 10 23:07:44.194854 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:44.194824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" event={"ID":"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44","Type":"ContainerStarted","Data":"b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c"} Jul 10 23:07:44.195007 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:44.194875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" event={"ID":"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44","Type":"ContainerStarted","Data":"2b2f65b766bd4c62baf6aef08cc7df84852ae2b41304ad0a44e0639ff899f19d"} Jul 10 23:07:44.195007 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:44.194926 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:44.217261 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:44.217218 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" podStartSLOduration=1.8726588450000001 podStartE2EDuration="2.217203103s" totalImagesPullingTime="344.544258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:07:44.214863562 +0000 UTC m=+1483.092555210" watchObservedRunningTime="2026-07-10 23:07:44.217203103 +0000 UTC m=+1483.094894722" Jul 10 23:07:47.443031 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.443000 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg"] Jul 10 23:07:47.443517 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.443260 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" podUID="a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" containerName="maas-api" containerID="cri-o://b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c" gracePeriod=30 Jul 10 23:07:47.689795 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.689766 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:47.792810 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.792741 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-openshift-service-ca\") pod \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " Jul 10 23:07:47.792810 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.792777 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8w5s\" (UniqueName: \"kubernetes.io/projected/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-kube-api-access-k8w5s\") pod \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " Jul 10 23:07:47.792996 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.792822 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls\") pod \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\" (UID: \"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44\") " Jul 10 23:07:47.793186 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.793130 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" (UID: "a0e081a4-0d4f-4e40-bda4-652d5e4e6f44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:07:47.794865 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.794833 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" (UID: "a0e081a4-0d4f-4e40-bda4-652d5e4e6f44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:07:47.794963 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.794884 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-kube-api-access-k8w5s" (OuterVolumeSpecName: "kube-api-access-k8w5s") pod "a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" (UID: "a0e081a4-0d4f-4e40-bda4-652d5e4e6f44"). InnerVolumeSpecName "kube-api-access-k8w5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:07:47.894500 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.894478 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:47.894500 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.894500 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8w5s\" (UniqueName: \"kubernetes.io/projected/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-kube-api-access-k8w5s\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:47.894649 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:47.894509 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:07:48.217477 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.217443 2577 generic.go:356] "Generic (PLEG): container finished" podID="a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" containerID="b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c" exitCode=0 Jul 10 23:07:48.217634 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.217489 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" Jul 10 23:07:48.217634 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.217517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" event={"ID":"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44","Type":"ContainerDied","Data":"b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c"} Jul 10 23:07:48.217634 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.217570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg" event={"ID":"a0e081a4-0d4f-4e40-bda4-652d5e4e6f44","Type":"ContainerDied","Data":"2b2f65b766bd4c62baf6aef08cc7df84852ae2b41304ad0a44e0639ff899f19d"} Jul 10 23:07:48.217634 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.217601 2577 scope.go:117] "RemoveContainer" containerID="b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c" Jul 10 23:07:48.227192 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.227174 2577 scope.go:117] "RemoveContainer" containerID="b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c" Jul 10 23:07:48.227434 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:07:48.227418 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c\": container with ID starting with b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c not found: ID does not exist" containerID="b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c" Jul 10 23:07:48.227504 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.227441 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c"} err="failed to get container status \"b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c\": rpc error: code = NotFound desc = could not find container \"b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c\": container with ID starting with b21f59e9ce9aa7eaedcf7585dca7dec642bb0a0dbb558b28755ddb8412a06b1c not found: ID does not exist" Jul 10 23:07:48.241720 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.241693 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg"] Jul 10 23:07:48.246137 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:48.246116 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b379716f-5b774fc776-fbgqg"] Jul 10 23:07:49.636097 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:07:49.636064 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" path="/var/lib/kubelet/pods/a0e081a4-0d4f-4e40-bda4-652d5e4e6f44/volumes" Jul 10 23:08:01.639293 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:08:01.639263 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:08:01.641610 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:08:01.641584 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:09:32.456386 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.456355 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn"] Jul 10 23:09:32.456911 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.456815 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" containerName="maas-api" Jul 10 23:09:32.456911 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.456832 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" containerName="maas-api" Jul 10 23:09:32.457079 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.456918 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e081a4-0d4f-4e40-bda4-652d5e4e6f44" containerName="maas-api" Jul 10 23:09:32.460003 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.459966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.462722 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.462702 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-eb6601c6-serving-cert\"" Jul 10 23:09:32.472893 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.472872 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn"] Jul 10 23:09:32.517260 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.517238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.517400 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.517379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a017d69-cf7b-42b1-9360-d31d7e163efa-openshift-service-ca\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.517461 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.517425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/1a017d69-cf7b-42b1-9360-d31d7e163efa-kube-api-access-q5vsf\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.618061 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.618036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a017d69-cf7b-42b1-9360-d31d7e163efa-openshift-service-ca\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.618210 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.618072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/1a017d69-cf7b-42b1-9360-d31d7e163efa-kube-api-access-q5vsf\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.618210 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.618100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.618210 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:09:32.618187 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-serving-cert: secret "maas-api-e2e-mt-eb6601c6-serving-cert" not found Jul 10 23:09:32.618394 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:09:32.618243 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls podName:1a017d69-cf7b-42b1-9360-d31d7e163efa nodeName:}" failed. No retries permitted until 2026-07-10 23:09:33.118228681 +0000 UTC m=+1591.995920277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls") pod "maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" (UID: "1a017d69-cf7b-42b1-9360-d31d7e163efa") : secret "maas-api-e2e-mt-eb6601c6-serving-cert" not found Jul 10 23:09:32.618702 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.618678 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a017d69-cf7b-42b1-9360-d31d7e163efa-openshift-service-ca\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:32.627232 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:32.627206 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/1a017d69-cf7b-42b1-9360-d31d7e163efa-kube-api-access-q5vsf\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:33.123754 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:33.123713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:33.126229 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:33.126198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls\") pod \"maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:33.369921 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:33.369888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:33.518486 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:33.518462 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn"] Jul 10 23:09:33.520250 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:09:33.520220 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a017d69_cf7b_42b1_9360_d31d7e163efa.slice/crio-bf5abec56c7e940dfc26d43c675b5866c562b25b5e1848195ac6f78ca3a15473 WatchSource:0}: Error finding container bf5abec56c7e940dfc26d43c675b5866c562b25b5e1848195ac6f78ca3a15473: Status 404 returned error can't find the container with id bf5abec56c7e940dfc26d43c675b5866c562b25b5e1848195ac6f78ca3a15473 Jul 10 23:09:33.757690 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:33.757659 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" event={"ID":"1a017d69-cf7b-42b1-9360-d31d7e163efa","Type":"ContainerStarted","Data":"bf5abec56c7e940dfc26d43c675b5866c562b25b5e1848195ac6f78ca3a15473"} Jul 10 23:09:34.765239 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:34.765201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" event={"ID":"1a017d69-cf7b-42b1-9360-d31d7e163efa","Type":"ContainerStarted","Data":"201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e"} Jul 10 23:09:34.765609 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:34.765262 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:34.784559 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:34.784517 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" podStartSLOduration=2.3016910409999998 podStartE2EDuration="2.784500426s" totalImagesPullingTime="482.809385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:09:34.78291847 +0000 UTC m=+1593.660610119" watchObservedRunningTime="2026-07-10 23:09:34.784500426 +0000 UTC m=+1593.662192045" Jul 10 23:09:40.774726 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:40.774697 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:44.526192 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.526157 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn"] Jul 10 23:09:44.526681 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.526514 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" podUID="1a017d69-cf7b-42b1-9360-d31d7e163efa" containerName="maas-api" containerID="cri-o://201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e" gracePeriod=30 Jul 10 23:09:44.772259 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.772238 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:44.816472 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.816403 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a017d69-cf7b-42b1-9360-d31d7e163efa" containerID="201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e" exitCode=0 Jul 10 23:09:44.816472 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.816445 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" Jul 10 23:09:44.816640 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.816477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" event={"ID":"1a017d69-cf7b-42b1-9360-d31d7e163efa","Type":"ContainerDied","Data":"201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e"} Jul 10 23:09:44.816640 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.816523 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn" event={"ID":"1a017d69-cf7b-42b1-9360-d31d7e163efa","Type":"ContainerDied","Data":"bf5abec56c7e940dfc26d43c675b5866c562b25b5e1848195ac6f78ca3a15473"} Jul 10 23:09:44.816640 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.816553 2577 scope.go:117] "RemoveContainer" containerID="201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e" Jul 10 23:09:44.824816 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.824800 2577 scope.go:117] "RemoveContainer" containerID="201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e" Jul 10 23:09:44.825073 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:09:44.825053 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e\": container with ID starting with 201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e not found: ID does not exist" containerID="201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e" Jul 10 23:09:44.825126 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.825082 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e"} err="failed to get container status \"201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e\": rpc error: code = NotFound desc = could not find container \"201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e\": container with ID starting with 201c4c5ecaf49c005d78c9f6a2b391bc82ff5e2998c0f634a849d21a0d0fce1e not found: ID does not exist" Jul 10 23:09:44.929478 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.929456 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls\") pod \"1a017d69-cf7b-42b1-9360-d31d7e163efa\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " Jul 10 23:09:44.929587 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.929493 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/1a017d69-cf7b-42b1-9360-d31d7e163efa-kube-api-access-q5vsf\") pod \"1a017d69-cf7b-42b1-9360-d31d7e163efa\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " Jul 10 23:09:44.929587 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.929560 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a017d69-cf7b-42b1-9360-d31d7e163efa-openshift-service-ca\") pod \"1a017d69-cf7b-42b1-9360-d31d7e163efa\" (UID: \"1a017d69-cf7b-42b1-9360-d31d7e163efa\") " Jul 10 23:09:44.929945 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.929923 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a017d69-cf7b-42b1-9360-d31d7e163efa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a017d69-cf7b-42b1-9360-d31d7e163efa" (UID: "1a017d69-cf7b-42b1-9360-d31d7e163efa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:09:44.931502 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.931480 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a017d69-cf7b-42b1-9360-d31d7e163efa-kube-api-access-q5vsf" (OuterVolumeSpecName: "kube-api-access-q5vsf") pod "1a017d69-cf7b-42b1-9360-d31d7e163efa" (UID: "1a017d69-cf7b-42b1-9360-d31d7e163efa"). InnerVolumeSpecName "kube-api-access-q5vsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:09:44.931593 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:44.931523 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a017d69-cf7b-42b1-9360-d31d7e163efa" (UID: "1a017d69-cf7b-42b1-9360-d31d7e163efa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:09:45.031094 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:45.031070 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a017d69-cf7b-42b1-9360-d31d7e163efa-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:09:45.031094 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:45.031091 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vsf\" (UniqueName: \"kubernetes.io/projected/1a017d69-cf7b-42b1-9360-d31d7e163efa-kube-api-access-q5vsf\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:09:45.031264 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:45.031101 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a017d69-cf7b-42b1-9360-d31d7e163efa-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:09:45.141295 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:45.141241 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn"] Jul 10 23:09:45.144485 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:45.144464 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb6601c6-649669d48d-hgxvn"] Jul 10 23:09:45.634950 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:09:45.634914 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a017d69-cf7b-42b1-9360-d31d7e163efa" path="/var/lib/kubelet/pods/1a017d69-cf7b-42b1-9360-d31d7e163efa/volumes" Jul 10 23:10:13.712902 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.712870 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v"] Jul 10 23:10:13.713310 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.713250 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a017d69-cf7b-42b1-9360-d31d7e163efa" containerName="maas-api" Jul 10 23:10:13.713310 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.713263 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a017d69-cf7b-42b1-9360-d31d7e163efa" containerName="maas-api" Jul 10 23:10:13.713383 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.713314 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a017d69-cf7b-42b1-9360-d31d7e163efa" containerName="maas-api" Jul 10 23:10:13.716243 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.716224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.723511 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.723494 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ea0b66f6-serving-cert\"" Jul 10 23:10:13.735189 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.735167 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v"] Jul 10 23:10:13.767239 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.767211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ccc9c-e0ac-4335-8777-164e406a98cf-openshift-service-ca\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.767347 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.767242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dkh9\" (UniqueName: \"kubernetes.io/projected/f76ccc9c-e0ac-4335-8777-164e406a98cf-kube-api-access-4dkh9\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.767347 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.767327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ccc9c-e0ac-4335-8777-164e406a98cf-maas-api-tls\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.868222 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.868188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ccc9c-e0ac-4335-8777-164e406a98cf-openshift-service-ca\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.868222 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.868222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkh9\" (UniqueName: \"kubernetes.io/projected/f76ccc9c-e0ac-4335-8777-164e406a98cf-kube-api-access-4dkh9\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.868459 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.868303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ccc9c-e0ac-4335-8777-164e406a98cf-maas-api-tls\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.869037 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.869010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ccc9c-e0ac-4335-8777-164e406a98cf-openshift-service-ca\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.870728 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.870709 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ccc9c-e0ac-4335-8777-164e406a98cf-maas-api-tls\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:13.877586 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:13.877559 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkh9\" (UniqueName: \"kubernetes.io/projected/f76ccc9c-e0ac-4335-8777-164e406a98cf-kube-api-access-4dkh9\") pod \"maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:14.026750 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:14.026679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:14.161776 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:14.161749 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v"] Jul 10 23:10:14.163358 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:10:14.163327 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76ccc9c_e0ac_4335_8777_164e406a98cf.slice/crio-f63d292da5ff6268c2fe79bfee14c1b59c0bd760f5c5b0749fa9f8aad7ad8422 WatchSource:0}: Error finding container f63d292da5ff6268c2fe79bfee14c1b59c0bd760f5c5b0749fa9f8aad7ad8422: Status 404 returned error can't find the container with id f63d292da5ff6268c2fe79bfee14c1b59c0bd760f5c5b0749fa9f8aad7ad8422 Jul 10 23:10:14.974425 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:14.974389 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" event={"ID":"f76ccc9c-e0ac-4335-8777-164e406a98cf","Type":"ContainerStarted","Data":"34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743"} Jul 10 23:10:14.974787 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:14.974442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" event={"ID":"f76ccc9c-e0ac-4335-8777-164e406a98cf","Type":"ContainerStarted","Data":"f63d292da5ff6268c2fe79bfee14c1b59c0bd760f5c5b0749fa9f8aad7ad8422"} Jul 10 23:10:14.974787 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:14.974489 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:10:14.994208 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:14.994167 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" podStartSLOduration=1.6285133250000001 podStartE2EDuration="1.994153209s" totalImagesPullingTime="365.639884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:10:14.992289419 +0000 UTC m=+1633.869981060" watchObservedRunningTime="2026-07-10 23:10:14.994153209 +0000 UTC m=+1633.871844827" Jul 10 23:10:20.985690 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:10:20.985665 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:11:06.456380 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.456346 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v"] Jul 10 23:11:06.457023 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.456574 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" podUID="f76ccc9c-e0ac-4335-8777-164e406a98cf" containerName="maas-api" containerID="cri-o://34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743" gracePeriod=30 Jul 10 23:11:06.712774 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.712721 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:11:06.826067 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.826042 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ccc9c-e0ac-4335-8777-164e406a98cf-maas-api-tls\") pod \"f76ccc9c-e0ac-4335-8777-164e406a98cf\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " Jul 10 23:11:06.826219 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.826124 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dkh9\" (UniqueName: \"kubernetes.io/projected/f76ccc9c-e0ac-4335-8777-164e406a98cf-kube-api-access-4dkh9\") pod \"f76ccc9c-e0ac-4335-8777-164e406a98cf\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " Jul 10 23:11:06.826279 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.826213 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ccc9c-e0ac-4335-8777-164e406a98cf-openshift-service-ca\") pod \"f76ccc9c-e0ac-4335-8777-164e406a98cf\" (UID: \"f76ccc9c-e0ac-4335-8777-164e406a98cf\") " Jul 10 23:11:06.826532 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.826511 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76ccc9c-e0ac-4335-8777-164e406a98cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f76ccc9c-e0ac-4335-8777-164e406a98cf" (UID: "f76ccc9c-e0ac-4335-8777-164e406a98cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:11:06.828065 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.828042 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76ccc9c-e0ac-4335-8777-164e406a98cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f76ccc9c-e0ac-4335-8777-164e406a98cf" (UID: "f76ccc9c-e0ac-4335-8777-164e406a98cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:11:06.828167 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.828097 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76ccc9c-e0ac-4335-8777-164e406a98cf-kube-api-access-4dkh9" (OuterVolumeSpecName: "kube-api-access-4dkh9") pod "f76ccc9c-e0ac-4335-8777-164e406a98cf" (UID: "f76ccc9c-e0ac-4335-8777-164e406a98cf"). InnerVolumeSpecName "kube-api-access-4dkh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:11:06.927452 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.927416 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dkh9\" (UniqueName: \"kubernetes.io/projected/f76ccc9c-e0ac-4335-8777-164e406a98cf-kube-api-access-4dkh9\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:11:06.927452 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.927443 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ccc9c-e0ac-4335-8777-164e406a98cf-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:11:06.927452 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:06.927453 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ccc9c-e0ac-4335-8777-164e406a98cf-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:11:07.238530 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.238497 2577 generic.go:356] "Generic (PLEG): container finished" podID="f76ccc9c-e0ac-4335-8777-164e406a98cf" containerID="34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743" exitCode=0 Jul 10 23:11:07.238702 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.238557 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" Jul 10 23:11:07.238702 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.238584 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" event={"ID":"f76ccc9c-e0ac-4335-8777-164e406a98cf","Type":"ContainerDied","Data":"34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743"} Jul 10 23:11:07.238702 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.238639 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v" event={"ID":"f76ccc9c-e0ac-4335-8777-164e406a98cf","Type":"ContainerDied","Data":"f63d292da5ff6268c2fe79bfee14c1b59c0bd760f5c5b0749fa9f8aad7ad8422"} Jul 10 23:11:07.238702 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.238662 2577 scope.go:117] "RemoveContainer" containerID="34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743" Jul 10 23:11:07.248387 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.248368 2577 scope.go:117] "RemoveContainer" containerID="34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743" Jul 10 23:11:07.248637 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:11:07.248619 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743\": container with ID starting with 34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743 not found: ID does not exist" containerID="34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743" Jul 10 23:11:07.248685 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.248645 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743"} err="failed to get container status \"34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743\": rpc error: code = NotFound desc = could not find container \"34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743\": container with ID starting with 34fe38760c0f6d8d1773a28ee71fb56d1a2446b2ef0d5675e3cfa02f13454743 not found: ID does not exist" Jul 10 23:11:07.265689 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.265671 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v"] Jul 10 23:11:07.267981 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.267940 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea0b66f6-84476fc749-rlc2v"] Jul 10 23:11:07.635255 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:11:07.635147 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76ccc9c-e0ac-4335-8777-164e406a98cf" path="/var/lib/kubelet/pods/f76ccc9c-e0ac-4335-8777-164e406a98cf/volumes" Jul 10 23:12:58.531291 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.531214 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96"] Jul 10 23:12:58.531837 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.531802 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76ccc9c-e0ac-4335-8777-164e406a98cf" containerName="maas-api" Jul 10 23:12:58.531837 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.531821 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76ccc9c-e0ac-4335-8777-164e406a98cf" containerName="maas-api" Jul 10 23:12:58.531960 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.531919 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76ccc9c-e0ac-4335-8777-164e406a98cf" containerName="maas-api" Jul 10 23:12:58.534999 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.534960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.537959 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.537935 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1dc3cf-serving-cert\"" Jul 10 23:12:58.547179 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.547157 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96"] Jul 10 23:12:58.680929 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.680900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnshq\" (UniqueName: \"kubernetes.io/projected/0f0c101d-9250-4228-b239-683a41b4d563-kube-api-access-mnshq\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.681109 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.680949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0c101d-9250-4228-b239-683a41b4d563-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.681109 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.681057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0c101d-9250-4228-b239-683a41b4d563-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.781569 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.781480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0c101d-9250-4228-b239-683a41b4d563-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.781740 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.781577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0c101d-9250-4228-b239-683a41b4d563-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.781740 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.781606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnshq\" (UniqueName: \"kubernetes.io/projected/0f0c101d-9250-4228-b239-683a41b4d563-kube-api-access-mnshq\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.782439 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.782413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0c101d-9250-4228-b239-683a41b4d563-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.783944 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.783922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0c101d-9250-4228-b239-683a41b4d563-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.793249 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.793225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnshq\" (UniqueName: \"kubernetes.io/projected/0f0c101d-9250-4228-b239-683a41b4d563-kube-api-access-mnshq\") pod \"maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.846531 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.846505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:58.980730 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.980697 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96"] Jul 10 23:12:58.982540 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:12:58.982510 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0c101d_9250_4228_b239_683a41b4d563.slice/crio-b1a4af8173cbd2b390bf1f574a2d23d8562ae2d6b1556bf79cf8b5ababfe94e3 WatchSource:0}: Error finding container b1a4af8173cbd2b390bf1f574a2d23d8562ae2d6b1556bf79cf8b5ababfe94e3: Status 404 returned error can't find the container with id b1a4af8173cbd2b390bf1f574a2d23d8562ae2d6b1556bf79cf8b5ababfe94e3 Jul 10 23:12:58.983824 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:58.983807 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 23:12:59.832093 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:59.832015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" event={"ID":"0f0c101d-9250-4228-b239-683a41b4d563","Type":"ContainerStarted","Data":"32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad"} Jul 10 23:12:59.832093 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:59.832070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" event={"ID":"0f0c101d-9250-4228-b239-683a41b4d563","Type":"ContainerStarted","Data":"b1a4af8173cbd2b390bf1f574a2d23d8562ae2d6b1556bf79cf8b5ababfe94e3"} Jul 10 23:12:59.832471 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:59.832119 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:12:59.853159 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:12:59.853116 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" podStartSLOduration=1.403549135 podStartE2EDuration="1.853101887s" totalImagesPullingTime="449.552752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:12:59.850782866 +0000 UTC m=+1798.728474521" watchObservedRunningTime="2026-07-10 23:12:59.853101887 +0000 UTC m=+1798.730793505" Jul 10 23:13:01.672700 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:01.672676 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:13:01.676046 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:01.676028 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:13:05.844564 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:05.844534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:13:06.576114 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.576082 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht"] Jul 10 23:13:06.580384 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.580360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.583102 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.583074 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-51f2b4-serving-cert\"" Jul 10 23:13:06.588087 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.588030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht"] Jul 10 23:13:06.752449 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.752412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.752622 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.752493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.752622 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.752518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdns\" (UniqueName: \"kubernetes.io/projected/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-kube-api-access-6rdns\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.853784 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.853696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.853784 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.853733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdns\" (UniqueName: \"kubernetes.io/projected/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-kube-api-access-6rdns\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.854315 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.853818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.854315 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:13:06.853867 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-serving-cert: secret "maas-api-e2e-inf-b-51f2b4-serving-cert" not found Jul 10 23:13:06.854315 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:13:06.853945 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls podName:5bf0b63b-e706-47ac-b782-7dfadf0ccdb5 nodeName:}" failed. No retries permitted until 2026-07-10 23:13:07.353924732 +0000 UTC m=+1806.231616334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls") pod "maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" (UID: "5bf0b63b-e706-47ac-b782-7dfadf0ccdb5") : secret "maas-api-e2e-inf-b-51f2b4-serving-cert" not found Jul 10 23:13:06.854517 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.854498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:06.865306 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:06.865286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdns\" (UniqueName: \"kubernetes.io/projected/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-kube-api-access-6rdns\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:07.358312 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:07.358282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:07.360770 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:07.360742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls\") pod \"maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:07.492485 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:07.492448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:07.628153 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:07.628131 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht"] Jul 10 23:13:07.629755 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:13:07.629721 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf0b63b_e706_47ac_b782_7dfadf0ccdb5.slice/crio-ea69118121437d2078cd4fdf19152a22cf87e17e4a773e4fecaedfce890b3eb6 WatchSource:0}: Error finding container ea69118121437d2078cd4fdf19152a22cf87e17e4a773e4fecaedfce890b3eb6: Status 404 returned error can't find the container with id ea69118121437d2078cd4fdf19152a22cf87e17e4a773e4fecaedfce890b3eb6 Jul 10 23:13:07.877730 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:07.877636 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" event={"ID":"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5","Type":"ContainerStarted","Data":"ea69118121437d2078cd4fdf19152a22cf87e17e4a773e4fecaedfce890b3eb6"} Jul 10 23:13:08.884186 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:08.884152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" event={"ID":"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5","Type":"ContainerStarted","Data":"7982347e4d9122e5b1484f39da8fd5ac3532de8426291c595942c61d65ab1650"} Jul 10 23:13:08.884648 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:08.884287 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:13:08.905369 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:08.905324 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" podStartSLOduration=2.559237778 podStartE2EDuration="2.905309694s" totalImagesPullingTime="346.071916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:13:08.902023387 +0000 UTC m=+1807.779715027" watchObservedRunningTime="2026-07-10 23:13:08.905309694 +0000 UTC m=+1807.783001313" Jul 10 23:13:14.893928 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:13:14.893902 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:14:26.787089 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:26.787046 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96"] Jul 10 23:14:26.787606 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:26.787279 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" podUID="0f0c101d-9250-4228-b239-683a41b4d563" containerName="maas-api" containerID="cri-o://32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad" gracePeriod=30 Jul 10 23:14:27.232382 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.232353 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:14:27.294847 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.294810 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f0c101d-9250-4228-b239-683a41b4d563" containerID="32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad" exitCode=0 Jul 10 23:14:27.295048 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.294858 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" Jul 10 23:14:27.295048 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.294857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" event={"ID":"0f0c101d-9250-4228-b239-683a41b4d563","Type":"ContainerDied","Data":"32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad"} Jul 10 23:14:27.295048 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.294913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96" event={"ID":"0f0c101d-9250-4228-b239-683a41b4d563","Type":"ContainerDied","Data":"b1a4af8173cbd2b390bf1f574a2d23d8562ae2d6b1556bf79cf8b5ababfe94e3"} Jul 10 23:14:27.295048 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.294943 2577 scope.go:117] "RemoveContainer" containerID="32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad" Jul 10 23:14:27.303542 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.303525 2577 scope.go:117] "RemoveContainer" containerID="32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad" Jul 10 23:14:27.303781 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:14:27.303762 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad\": container with ID starting with 32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad not found: ID does not exist" containerID="32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad" Jul 10 23:14:27.303822 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.303791 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad"} err="failed to get container status \"32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad\": rpc error: code = NotFound desc = could not find container \"32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad\": container with ID starting with 32dbe4741ace6a04ffdfec3396f3db75506efd7ed9782ea3161b4b1298f09dad not found: ID does not exist" Jul 10 23:14:27.341573 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.341511 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0c101d-9250-4228-b239-683a41b4d563-openshift-service-ca\") pod \"0f0c101d-9250-4228-b239-683a41b4d563\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " Jul 10 23:14:27.341573 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.341547 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0c101d-9250-4228-b239-683a41b4d563-maas-api-tls\") pod \"0f0c101d-9250-4228-b239-683a41b4d563\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " Jul 10 23:14:27.341573 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.341571 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnshq\" (UniqueName: \"kubernetes.io/projected/0f0c101d-9250-4228-b239-683a41b4d563-kube-api-access-mnshq\") pod \"0f0c101d-9250-4228-b239-683a41b4d563\" (UID: \"0f0c101d-9250-4228-b239-683a41b4d563\") " Jul 10 23:14:27.341985 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.341947 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0c101d-9250-4228-b239-683a41b4d563-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f0c101d-9250-4228-b239-683a41b4d563" (UID: "0f0c101d-9250-4228-b239-683a41b4d563"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:14:27.343616 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.343583 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0c101d-9250-4228-b239-683a41b4d563-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f0c101d-9250-4228-b239-683a41b4d563" (UID: "0f0c101d-9250-4228-b239-683a41b4d563"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:14:27.343700 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.343647 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0c101d-9250-4228-b239-683a41b4d563-kube-api-access-mnshq" (OuterVolumeSpecName: "kube-api-access-mnshq") pod "0f0c101d-9250-4228-b239-683a41b4d563" (UID: "0f0c101d-9250-4228-b239-683a41b4d563"). InnerVolumeSpecName "kube-api-access-mnshq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:14:27.443051 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.443005 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0c101d-9250-4228-b239-683a41b4d563-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:14:27.443051 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.443045 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0c101d-9250-4228-b239-683a41b4d563-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:14:27.443051 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.443057 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnshq\" (UniqueName: \"kubernetes.io/projected/0f0c101d-9250-4228-b239-683a41b4d563-kube-api-access-mnshq\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:14:27.617349 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.617320 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96"] Jul 10 23:14:27.621300 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.621275 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc3cf-667d84849c-zql96"] Jul 10 23:14:27.635739 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:14:27.635718 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f0c101d-9250-4228-b239-683a41b4d563" path="/var/lib/kubelet/pods/0f0c101d-9250-4228-b239-683a41b4d563/volumes" Jul 10 23:15:00.143566 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.143531 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh"] Jul 10 23:15:00.144167 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.144092 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0c101d-9250-4228-b239-683a41b4d563" containerName="maas-api" Jul 10 23:15:00.144167 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.144110 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0c101d-9250-4228-b239-683a41b4d563" containerName="maas-api" Jul 10 23:15:00.144277 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.144228 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0c101d-9250-4228-b239-683a41b4d563" containerName="maas-api" Jul 10 23:15:00.147253 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.147235 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2"] Jul 10 23:15:00.147367 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.147348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" Jul 10 23:15:00.150135 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.150121 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dh5r5\"" Jul 10 23:15:00.150470 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.150457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" Jul 10 23:15:00.156192 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.156171 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh"] Jul 10 23:15:00.159351 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.159332 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2"] Jul 10 23:15:00.159754 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.159735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" Jul 10 23:15:00.168184 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.168168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" Jul 10 23:15:00.304379 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.304351 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh"] Jul 10 23:15:00.306182 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:15:00.306152 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6809772f_0da0_44af_9450_981d7f574da7.slice/crio-bbaddaa406c7910dd1ba94a505ded4601d5395b5c8e91d4ec007e83470bc11d0 WatchSource:0}: Error finding container bbaddaa406c7910dd1ba94a505ded4601d5395b5c8e91d4ec007e83470bc11d0: Status 404 returned error can't find the container with id bbaddaa406c7910dd1ba94a505ded4601d5395b5c8e91d4ec007e83470bc11d0 Jul 10 23:15:00.337125 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.337103 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2"] Jul 10 23:15:00.341045 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:15:00.341015 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f191663_9019_40f2_ae81_53508d9acac9.slice/crio-67601f0c812161a4a8926bc00ebfdabba8d32bd633be3e8217c0fb962e0e8b29 WatchSource:0}: Error finding container 67601f0c812161a4a8926bc00ebfdabba8d32bd633be3e8217c0fb962e0e8b29: Status 404 returned error can't find the container with id 67601f0c812161a4a8926bc00ebfdabba8d32bd633be3e8217c0fb962e0e8b29 Jul 10 23:15:00.467171 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.467147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" event={"ID":"5f191663-9019-40f2-ae81-53508d9acac9","Type":"ContainerStarted","Data":"67601f0c812161a4a8926bc00ebfdabba8d32bd633be3e8217c0fb962e0e8b29"} Jul 10 23:15:00.468329 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:00.468310 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" event={"ID":"6809772f-0da0-44af-9450-981d7f574da7","Type":"ContainerStarted","Data":"bbaddaa406c7910dd1ba94a505ded4601d5395b5c8e91d4ec007e83470bc11d0"} Jul 10 23:15:01.475392 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:01.475361 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" event={"ID":"5f191663-9019-40f2-ae81-53508d9acac9","Type":"ContainerStarted","Data":"239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c"} Jul 10 23:15:01.476818 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:01.476790 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" event={"ID":"6809772f-0da0-44af-9450-981d7f574da7","Type":"ContainerStarted","Data":"51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a"} Jul 10 23:15:01.493000 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:01.492940 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" podStartSLOduration=1.4929270159999999 podStartE2EDuration="1.492927016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:15:01.49182487 +0000 UTC m=+1920.369516509" watchObservedRunningTime="2026-07-10 23:15:01.492927016 +0000 UTC m=+1920.370618612" Jul 10 23:15:01.509198 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:01.509159 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" podStartSLOduration=1.509143587 podStartE2EDuration="1.509143587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:15:01.506632775 +0000 UTC m=+1920.384324417" watchObservedRunningTime="2026-07-10 23:15:01.509143587 +0000 UTC m=+1920.386835205" Jul 10 23:15:10.133934 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.133903 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht"] Jul 10 23:15:10.134460 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.134155 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" podUID="5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" containerName="maas-api" containerID="cri-o://7982347e4d9122e5b1484f39da8fd5ac3532de8426291c595942c61d65ab1650" gracePeriod=30 Jul 10 23:15:10.172841 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.172801 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh"] Jul 10 23:15:10.173046 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.173025 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" podUID="6809772f-0da0-44af-9450-981d7f574da7" containerName="cleanup" containerID="cri-o://51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a" gracePeriod=30 Jul 10 23:15:10.524107 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.524079 2577 generic.go:356] "Generic (PLEG): container finished" podID="5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" containerID="7982347e4d9122e5b1484f39da8fd5ac3532de8426291c595942c61d65ab1650" exitCode=0 Jul 10 23:15:10.524260 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.524132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" event={"ID":"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5","Type":"ContainerDied","Data":"7982347e4d9122e5b1484f39da8fd5ac3532de8426291c595942c61d65ab1650"} Jul 10 23:15:10.579468 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.579444 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:15:10.613676 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.613655 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rdns\" (UniqueName: \"kubernetes.io/projected/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-kube-api-access-6rdns\") pod \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " Jul 10 23:15:10.613838 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.613719 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-openshift-service-ca\") pod \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " Jul 10 23:15:10.613838 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.613779 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls\") pod \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\" (UID: \"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5\") " Jul 10 23:15:10.614063 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.614041 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" (UID: "5bf0b63b-e706-47ac-b782-7dfadf0ccdb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:15:10.615828 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.615796 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" (UID: "5bf0b63b-e706-47ac-b782-7dfadf0ccdb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:15:10.615918 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.615854 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-kube-api-access-6rdns" (OuterVolumeSpecName: "kube-api-access-6rdns") pod "5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" (UID: "5bf0b63b-e706-47ac-b782-7dfadf0ccdb5"). InnerVolumeSpecName "kube-api-access-6rdns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:15:10.714760 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.714697 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:10.714760 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.714719 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:10.714760 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:10.714729 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rdns\" (UniqueName: \"kubernetes.io/projected/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5-kube-api-access-6rdns\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:11.530738 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:11.530706 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" Jul 10 23:15:11.530738 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:11.530722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht" event={"ID":"5bf0b63b-e706-47ac-b782-7dfadf0ccdb5","Type":"ContainerDied","Data":"ea69118121437d2078cd4fdf19152a22cf87e17e4a773e4fecaedfce890b3eb6"} Jul 10 23:15:11.531253 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:11.530776 2577 scope.go:117] "RemoveContainer" containerID="7982347e4d9122e5b1484f39da8fd5ac3532de8426291c595942c61d65ab1650" Jul 10 23:15:11.552581 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:11.552556 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht"] Jul 10 23:15:11.557178 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:11.557157 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-51f2b4-6d96cb4f56-ggjht"] Jul 10 23:15:11.635561 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:11.635536 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" path="/var/lib/kubelet/pods/5bf0b63b-e706-47ac-b782-7dfadf0ccdb5/volumes" Jul 10 23:15:19.291219 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.291186 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n"] Jul 10 23:15:19.291720 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.291702 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" containerName="maas-api" Jul 10 23:15:19.291765 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.291723 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" containerName="maas-api" Jul 10 23:15:19.291796 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.291784 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf0b63b-e706-47ac-b782-7dfadf0ccdb5" containerName="maas-api" Jul 10 23:15:19.296101 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.296076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.298840 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.298822 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d639f4-serving-cert\"" Jul 10 23:15:19.308111 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.308087 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n"] Jul 10 23:15:19.388766 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.388736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr59k\" (UniqueName: \"kubernetes.io/projected/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-kube-api-access-qr59k\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.393025 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.388773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-openshift-service-ca\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.393025 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.389390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-maas-api-tls\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.491080 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.491047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr59k\" (UniqueName: \"kubernetes.io/projected/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-kube-api-access-qr59k\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.491240 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.491088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-openshift-service-ca\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.491240 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.491130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-maas-api-tls\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.491855 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.491834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-openshift-service-ca\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.496146 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.495941 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-maas-api-tls\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.500283 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.500261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr59k\" (UniqueName: \"kubernetes.io/projected/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-kube-api-access-qr59k\") pod \"maas-api-e2e-api-a-d639f4-79cd58c74-5s72n\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.608086 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.608018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:19.746446 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:19.746410 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n"] Jul 10 23:15:19.748174 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:15:19.748146 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9c5c75_dcb9_46c1_8cf1_a3481765566d.slice/crio-5503753e9c0e2dffed521a21c70fa52dbd789b884c1e63e5b044bf79187cf6ca WatchSource:0}: Error finding container 5503753e9c0e2dffed521a21c70fa52dbd789b884c1e63e5b044bf79187cf6ca: Status 404 returned error can't find the container with id 5503753e9c0e2dffed521a21c70fa52dbd789b884c1e63e5b044bf79187cf6ca Jul 10 23:15:20.581455 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:20.581371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" event={"ID":"ce9c5c75-dcb9-46c1-8cf1-a3481765566d","Type":"ContainerStarted","Data":"da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b"} Jul 10 23:15:20.581455 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:20.581436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" event={"ID":"ce9c5c75-dcb9-46c1-8cf1-a3481765566d","Type":"ContainerStarted","Data":"5503753e9c0e2dffed521a21c70fa52dbd789b884c1e63e5b044bf79187cf6ca"} Jul 10 23:15:20.581837 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:20.581535 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:20.599281 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:20.599236 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" podStartSLOduration=1.193455341 podStartE2EDuration="1.599219831s" totalImagesPullingTime="405.76449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:15:20.598144783 +0000 UTC m=+1939.475836426" watchObservedRunningTime="2026-07-10 23:15:20.599219831 +0000 UTC m=+1939.476911448" Jul 10 23:15:26.592164 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:26.592140 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:27.360473 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.360441 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7"] Jul 10 23:15:27.364154 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.364121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.367413 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.367392 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8b0687-serving-cert\"" Jul 10 23:15:27.378225 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.378204 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7"] Jul 10 23:15:27.464577 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.464544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.464723 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.464583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnsm\" (UniqueName: \"kubernetes.io/projected/545ed8bd-34fa-46c7-8721-ea50508e4e75-kube-api-access-mmnsm\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.464723 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.464675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545ed8bd-34fa-46c7-8721-ea50508e4e75-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.565210 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.565174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.565210 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.565215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnsm\" (UniqueName: \"kubernetes.io/projected/545ed8bd-34fa-46c7-8721-ea50508e4e75-kube-api-access-mmnsm\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.565501 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.565298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545ed8bd-34fa-46c7-8721-ea50508e4e75-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.565501 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:15:27.565316 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-serving-cert: secret "maas-api-e2e-api-b-8b0687-serving-cert" not found Jul 10 23:15:27.565501 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:15:27.565381 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls podName:545ed8bd-34fa-46c7-8721-ea50508e4e75 nodeName:}" failed. No retries permitted until 2026-07-10 23:15:28.065360707 +0000 UTC m=+1946.943052303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls") pod "maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" (UID: "545ed8bd-34fa-46c7-8721-ea50508e4e75") : secret "maas-api-e2e-api-b-8b0687-serving-cert" not found Jul 10 23:15:27.566150 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.566129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545ed8bd-34fa-46c7-8721-ea50508e4e75-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:27.574112 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:27.574084 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnsm\" (UniqueName: \"kubernetes.io/projected/545ed8bd-34fa-46c7-8721-ea50508e4e75-kube-api-access-mmnsm\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:28.071108 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:28.071080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:28.073397 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:28.073378 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls\") pod \"maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:28.275104 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:28.275072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:28.423965 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:28.423944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7"] Jul 10 23:15:28.425511 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:15:28.425484 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545ed8bd_34fa_46c7_8721_ea50508e4e75.slice/crio-8b7a0bd172256636d67d842d355c0c2cb892291a80104e2d437d9c79e665efbe WatchSource:0}: Error finding container 8b7a0bd172256636d67d842d355c0c2cb892291a80104e2d437d9c79e665efbe: Status 404 returned error can't find the container with id 8b7a0bd172256636d67d842d355c0c2cb892291a80104e2d437d9c79e665efbe Jul 10 23:15:28.624742 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:28.624660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" event={"ID":"545ed8bd-34fa-46c7-8721-ea50508e4e75","Type":"ContainerStarted","Data":"8b7a0bd172256636d67d842d355c0c2cb892291a80104e2d437d9c79e665efbe"} Jul 10 23:15:29.634895 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:29.634868 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:29.635323 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:29.634913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" event={"ID":"545ed8bd-34fa-46c7-8721-ea50508e4e75","Type":"ContainerStarted","Data":"b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a"} Jul 10 23:15:29.654240 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:29.654195 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" podStartSLOduration=2.295450292 podStartE2EDuration="2.654180452s" totalImagesPullingTime="358.73016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:15:29.651948125 +0000 UTC m=+1948.529639764" watchObservedRunningTime="2026-07-10 23:15:29.654180452 +0000 UTC m=+1948.531872071" Jul 10 23:15:35.642843 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:35.642809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:39.824225 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:39.824185 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n"] Jul 10 23:15:39.824641 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:39.824436 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" podUID="ce9c5c75-dcb9-46c1-8cf1-a3481765566d" containerName="maas-api" containerID="cri-o://da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b" gracePeriod=30 Jul 10 23:15:40.074832 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.074778 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:40.181693 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.181669 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-openshift-service-ca\") pod \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " Jul 10 23:15:40.181825 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.181736 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-maas-api-tls\") pod \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " Jul 10 23:15:40.181825 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.181784 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr59k\" (UniqueName: \"kubernetes.io/projected/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-kube-api-access-qr59k\") pod \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\" (UID: \"ce9c5c75-dcb9-46c1-8cf1-a3481765566d\") " Jul 10 23:15:40.182054 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.182032 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce9c5c75-dcb9-46c1-8cf1-a3481765566d" (UID: "ce9c5c75-dcb9-46c1-8cf1-a3481765566d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:15:40.183746 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.183725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce9c5c75-dcb9-46c1-8cf1-a3481765566d" (UID: "ce9c5c75-dcb9-46c1-8cf1-a3481765566d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:15:40.183814 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.183782 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-kube-api-access-qr59k" (OuterVolumeSpecName: "kube-api-access-qr59k") pod "ce9c5c75-dcb9-46c1-8cf1-a3481765566d" (UID: "ce9c5c75-dcb9-46c1-8cf1-a3481765566d"). InnerVolumeSpecName "kube-api-access-qr59k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:15:40.282882 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.282858 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:40.282882 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.282882 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:40.283102 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.282891 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr59k\" (UniqueName: \"kubernetes.io/projected/ce9c5c75-dcb9-46c1-8cf1-a3481765566d-kube-api-access-qr59k\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:40.374615 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.374593 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" Jul 10 23:15:40.690088 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.690051 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce9c5c75-dcb9-46c1-8cf1-a3481765566d" containerID="da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b" exitCode=0 Jul 10 23:15:40.690268 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.690102 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" Jul 10 23:15:40.690268 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.690133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" event={"ID":"ce9c5c75-dcb9-46c1-8cf1-a3481765566d","Type":"ContainerDied","Data":"da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b"} Jul 10 23:15:40.690268 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.690185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n" event={"ID":"ce9c5c75-dcb9-46c1-8cf1-a3481765566d","Type":"ContainerDied","Data":"5503753e9c0e2dffed521a21c70fa52dbd789b884c1e63e5b044bf79187cf6ca"} Jul 10 23:15:40.690268 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.690208 2577 scope.go:117] "RemoveContainer" containerID="da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b" Jul 10 23:15:40.691526 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.691508 2577 generic.go:356] "Generic (PLEG): container finished" podID="6809772f-0da0-44af-9450-981d7f574da7" containerID="51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a" exitCode=137 Jul 10 23:15:40.691635 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.691553 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" Jul 10 23:15:40.691635 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.691587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" event={"ID":"6809772f-0da0-44af-9450-981d7f574da7","Type":"ContainerDied","Data":"51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a"} Jul 10 23:15:40.691751 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.691638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh" event={"ID":"6809772f-0da0-44af-9450-981d7f574da7","Type":"ContainerDied","Data":"bbaddaa406c7910dd1ba94a505ded4601d5395b5c8e91d4ec007e83470bc11d0"} Jul 10 23:15:40.699550 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.699535 2577 scope.go:117] "RemoveContainer" containerID="da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b" Jul 10 23:15:40.699824 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:15:40.699803 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b\": container with ID starting with da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b not found: ID does not exist" containerID="da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b" Jul 10 23:15:40.699889 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.699838 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b"} err="failed to get container status \"da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b\": rpc error: code = NotFound desc = could not find container \"da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b\": container with ID starting with da4a17948f81ed3cce4a043e2a782a34456cfe191ada87ff2cba097ca3100f4b not found: ID does not exist" Jul 10 23:15:40.699889 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.699862 2577 scope.go:117] "RemoveContainer" containerID="51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a" Jul 10 23:15:40.707621 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.707605 2577 scope.go:117] "RemoveContainer" containerID="51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a" Jul 10 23:15:40.707845 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:15:40.707829 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a\": container with ID starting with 51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a not found: ID does not exist" containerID="51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a" Jul 10 23:15:40.707898 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.707853 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a"} err="failed to get container status \"51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a\": rpc error: code = NotFound desc = could not find container \"51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a\": container with ID starting with 51e63bd247d4d04abbf6063f026532dc737c72267af5a2842515d1872546d66a not found: ID does not exist" Jul 10 23:15:40.715270 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.715249 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh"] Jul 10 23:15:40.721064 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.721045 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-51f2b4-29728755-7g6mh"] Jul 10 23:15:40.732030 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.732012 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n"] Jul 10 23:15:40.735889 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:40.735868 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d639f4-79cd58c74-5s72n"] Jul 10 23:15:41.636948 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:41.636917 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6809772f-0da0-44af-9450-981d7f574da7" path="/var/lib/kubelet/pods/6809772f-0da0-44af-9450-981d7f574da7/volumes" Jul 10 23:15:41.637322 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:41.637307 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9c5c75-dcb9-46c1-8cf1-a3481765566d" path="/var/lib/kubelet/pods/ce9c5c75-dcb9-46c1-8cf1-a3481765566d/volumes" Jul 10 23:15:48.056667 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.056633 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7"] Jul 10 23:15:48.057462 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.057429 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" podUID="545ed8bd-34fa-46c7-8721-ea50508e4e75" containerName="maas-api" containerID="cri-o://b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a" gracePeriod=30 Jul 10 23:15:48.310617 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.310554 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:48.454367 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.454337 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls\") pod \"545ed8bd-34fa-46c7-8721-ea50508e4e75\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " Jul 10 23:15:48.454535 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.454387 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmnsm\" (UniqueName: \"kubernetes.io/projected/545ed8bd-34fa-46c7-8721-ea50508e4e75-kube-api-access-mmnsm\") pod \"545ed8bd-34fa-46c7-8721-ea50508e4e75\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " Jul 10 23:15:48.454535 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.454450 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545ed8bd-34fa-46c7-8721-ea50508e4e75-openshift-service-ca\") pod \"545ed8bd-34fa-46c7-8721-ea50508e4e75\" (UID: \"545ed8bd-34fa-46c7-8721-ea50508e4e75\") " Jul 10 23:15:48.454810 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.454781 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545ed8bd-34fa-46c7-8721-ea50508e4e75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "545ed8bd-34fa-46c7-8721-ea50508e4e75" (UID: "545ed8bd-34fa-46c7-8721-ea50508e4e75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:15:48.456448 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.456414 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "545ed8bd-34fa-46c7-8721-ea50508e4e75" (UID: "545ed8bd-34fa-46c7-8721-ea50508e4e75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:15:48.456552 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.456479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545ed8bd-34fa-46c7-8721-ea50508e4e75-kube-api-access-mmnsm" (OuterVolumeSpecName: "kube-api-access-mmnsm") pod "545ed8bd-34fa-46c7-8721-ea50508e4e75" (UID: "545ed8bd-34fa-46c7-8721-ea50508e4e75"). InnerVolumeSpecName "kube-api-access-mmnsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:15:48.555289 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.555260 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmnsm\" (UniqueName: \"kubernetes.io/projected/545ed8bd-34fa-46c7-8721-ea50508e4e75-kube-api-access-mmnsm\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:48.555289 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.555285 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545ed8bd-34fa-46c7-8721-ea50508e4e75-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:48.555289 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.555296 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545ed8bd-34fa-46c7-8721-ea50508e4e75-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:15:48.736811 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.736774 2577 generic.go:356] "Generic (PLEG): container finished" podID="545ed8bd-34fa-46c7-8721-ea50508e4e75" containerID="b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a" exitCode=0 Jul 10 23:15:48.737002 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.736837 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" Jul 10 23:15:48.737002 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.736857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" event={"ID":"545ed8bd-34fa-46c7-8721-ea50508e4e75","Type":"ContainerDied","Data":"b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a"} Jul 10 23:15:48.737002 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.736912 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7" event={"ID":"545ed8bd-34fa-46c7-8721-ea50508e4e75","Type":"ContainerDied","Data":"8b7a0bd172256636d67d842d355c0c2cb892291a80104e2d437d9c79e665efbe"} Jul 10 23:15:48.737002 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.736935 2577 scope.go:117] "RemoveContainer" containerID="b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a" Jul 10 23:15:48.748763 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.748748 2577 scope.go:117] "RemoveContainer" containerID="b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a" Jul 10 23:15:48.749333 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:15:48.749307 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a\": container with ID starting with b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a not found: ID does not exist" containerID="b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a" Jul 10 23:15:48.749436 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.749345 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a"} err="failed to get container status \"b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a\": rpc error: code = NotFound desc = could not find container \"b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a\": container with ID starting with b2133c58994f7efdf22a1d9b32eb064f8ef66db7b701e01608105acf7800512a not found: ID does not exist" Jul 10 23:15:48.759931 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.759911 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7"] Jul 10 23:15:48.766477 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:48.766456 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b0687-5dd859d6d4-472q7"] Jul 10 23:15:49.635488 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:49.635447 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545ed8bd-34fa-46c7-8721-ea50508e4e75" path="/var/lib/kubelet/pods/545ed8bd-34fa-46c7-8721-ea50508e4e75/volumes" Jul 10 23:15:57.555048 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.554935 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c"] Jul 10 23:15:57.555543 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555526 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545ed8bd-34fa-46c7-8721-ea50508e4e75" containerName="maas-api" Jul 10 23:15:57.555626 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555545 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="545ed8bd-34fa-46c7-8721-ea50508e4e75" containerName="maas-api" Jul 10 23:15:57.555626 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555575 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6809772f-0da0-44af-9450-981d7f574da7" containerName="cleanup" Jul 10 23:15:57.555626 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555584 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6809772f-0da0-44af-9450-981d7f574da7" containerName="cleanup" Jul 10 23:15:57.555626 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555601 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9c5c75-dcb9-46c1-8cf1-a3481765566d" containerName="maas-api" Jul 10 23:15:57.555626 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555610 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9c5c75-dcb9-46c1-8cf1-a3481765566d" containerName="maas-api" Jul 10 23:15:57.555874 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555710 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9c5c75-dcb9-46c1-8cf1-a3481765566d" containerName="maas-api" Jul 10 23:15:57.555874 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555722 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="545ed8bd-34fa-46c7-8721-ea50508e4e75" containerName="maas-api" Jul 10 23:15:57.555874 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.555734 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6809772f-0da0-44af-9450-981d7f574da7" containerName="cleanup" Jul 10 23:15:57.560189 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.560170 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.562641 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.562619 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-6d6384-serving-cert\"" Jul 10 23:15:57.570034 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.570010 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c"] Jul 10 23:15:57.629823 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.629790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.630059 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.629857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.630059 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.629955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbp8\" (UniqueName: \"kubernetes.io/projected/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-kube-api-access-cqbp8\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.730566 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.730524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.730758 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.730586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbp8\" (UniqueName: \"kubernetes.io/projected/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-kube-api-access-cqbp8\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.730758 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.730638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.730951 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:15:57.730903 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-serving-cert: secret "maas-api-e2e-shared-a-6d6384-serving-cert" not found Jul 10 23:15:57.731111 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:15:57.731029 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls podName:da1b77cb-c4e9-4e22-b0dc-e4ba0585e444 nodeName:}" failed. No retries permitted until 2026-07-10 23:15:58.23101163 +0000 UTC m=+1977.108703231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls") pod "maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" (UID: "da1b77cb-c4e9-4e22-b0dc-e4ba0585e444") : secret "maas-api-e2e-shared-a-6d6384-serving-cert" not found Jul 10 23:15:57.731216 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.731195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:57.741114 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:57.741089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbp8\" (UniqueName: \"kubernetes.io/projected/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-kube-api-access-cqbp8\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:58.234364 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:58.234320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:58.236854 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:58.236828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls\") pod \"maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:58.473032 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:58.472992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:58.635937 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:58.635904 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c"] Jul 10 23:15:58.636371 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:15:58.636341 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1b77cb_c4e9_4e22_b0dc_e4ba0585e444.slice/crio-7c87836e1120386b2f3f3f2ef8c786e4e8102c2f5d9d331544ddefdeca8b55b0 WatchSource:0}: Error finding container 7c87836e1120386b2f3f3f2ef8c786e4e8102c2f5d9d331544ddefdeca8b55b0: Status 404 returned error can't find the container with id 7c87836e1120386b2f3f3f2ef8c786e4e8102c2f5d9d331544ddefdeca8b55b0 Jul 10 23:15:58.795381 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:58.795286 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" event={"ID":"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444","Type":"ContainerStarted","Data":"7c87836e1120386b2f3f3f2ef8c786e4e8102c2f5d9d331544ddefdeca8b55b0"} Jul 10 23:15:59.802191 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:59.802153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" event={"ID":"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444","Type":"ContainerStarted","Data":"e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179"} Jul 10 23:15:59.802609 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:59.802228 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:15:59.826645 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:15:59.826597 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" podStartSLOduration=2.4880096800000002 podStartE2EDuration="2.826581607s" totalImagesPullingTime="338.571927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:15:59.823741468 +0000 UTC m=+1978.701433087" watchObservedRunningTime="2026-07-10 23:15:59.826581607 +0000 UTC m=+1978.704273226" Jul 10 23:16:00.016496 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:00.016459 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2"] Jul 10 23:16:00.016685 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:00.016664 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" podUID="5f191663-9019-40f2-ae81-53508d9acac9" containerName="cleanup" containerID="cri-o://239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c" gracePeriod=30 Jul 10 23:16:05.580928 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.580897 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh"] Jul 10 23:16:05.584529 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.584500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.587093 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.587061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-02c498-serving-cert\"" Jul 10 23:16:05.595965 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.595902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh"] Jul 10 23:16:05.707062 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.707027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.707062 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.707063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgzj\" (UniqueName: \"kubernetes.io/projected/06f3b422-a447-40c0-9a28-690805278fd6-kube-api-access-lwgzj\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.707321 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.707145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f3b422-a447-40c0-9a28-690805278fd6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.808026 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.807991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.808026 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.808027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgzj\" (UniqueName: \"kubernetes.io/projected/06f3b422-a447-40c0-9a28-690805278fd6-kube-api-access-lwgzj\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.808233 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.808052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f3b422-a447-40c0-9a28-690805278fd6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.808233 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:16:05.808140 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-serving-cert: secret "maas-api-e2e-shared-b-02c498-serving-cert" not found Jul 10 23:16:05.808233 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:16:05.808223 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls podName:06f3b422-a447-40c0-9a28-690805278fd6 nodeName:}" failed. No retries permitted until 2026-07-10 23:16:06.308200815 +0000 UTC m=+1985.185892411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls") pod "maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" (UID: "06f3b422-a447-40c0-9a28-690805278fd6") : secret "maas-api-e2e-shared-b-02c498-serving-cert" not found Jul 10 23:16:05.808666 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.808637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f3b422-a447-40c0-9a28-690805278fd6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:05.813686 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.813668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:16:05.820343 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:05.820316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgzj\" (UniqueName: \"kubernetes.io/projected/06f3b422-a447-40c0-9a28-690805278fd6-kube-api-access-lwgzj\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:06.313924 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:06.313894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:06.316275 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:06.316244 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls\") pod \"maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:06.496225 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:06.496188 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:06.639404 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:06.639373 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh"] Jul 10 23:16:06.640911 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:16:06.640870 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f3b422_a447_40c0_9a28_690805278fd6.slice/crio-5d2be4b2af95436dde7d80e680081a52fc63daadf09b89c17225d0fab9565985 WatchSource:0}: Error finding container 5d2be4b2af95436dde7d80e680081a52fc63daadf09b89c17225d0fab9565985: Status 404 returned error can't find the container with id 5d2be4b2af95436dde7d80e680081a52fc63daadf09b89c17225d0fab9565985 Jul 10 23:16:06.841604 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:06.841522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" event={"ID":"06f3b422-a447-40c0-9a28-690805278fd6","Type":"ContainerStarted","Data":"5d2be4b2af95436dde7d80e680081a52fc63daadf09b89c17225d0fab9565985"} Jul 10 23:16:07.849544 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:07.849507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" event={"ID":"06f3b422-a447-40c0-9a28-690805278fd6","Type":"ContainerStarted","Data":"9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10"} Jul 10 23:16:07.849960 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:07.849635 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:07.866728 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:07.866671 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" podStartSLOduration=2.41004421 podStartE2EDuration="2.866652664s" totalImagesPullingTime="456.608454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:16:07.866347386 +0000 UTC m=+1986.744039006" watchObservedRunningTime="2026-07-10 23:16:07.866652664 +0000 UTC m=+1986.744344284" Jul 10 23:16:13.860673 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:13.860640 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:16:30.190475 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.190449 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" Jul 10 23:16:30.974349 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.974307 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f191663-9019-40f2-ae81-53508d9acac9" containerID="239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c" exitCode=137 Jul 10 23:16:30.974533 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.974377 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" Jul 10 23:16:30.974533 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.974398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" event={"ID":"5f191663-9019-40f2-ae81-53508d9acac9","Type":"ContainerDied","Data":"239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c"} Jul 10 23:16:30.974533 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.974464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2" event={"ID":"5f191663-9019-40f2-ae81-53508d9acac9","Type":"ContainerDied","Data":"67601f0c812161a4a8926bc00ebfdabba8d32bd633be3e8217c0fb962e0e8b29"} Jul 10 23:16:30.974533 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.974487 2577 scope.go:117] "RemoveContainer" containerID="239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c" Jul 10 23:16:30.983209 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.983188 2577 scope.go:117] "RemoveContainer" containerID="239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c" Jul 10 23:16:30.983471 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:16:30.983452 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c\": container with ID starting with 239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c not found: ID does not exist" containerID="239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c" Jul 10 23:16:30.983522 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.983482 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c"} err="failed to get container status \"239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c\": rpc error: code = NotFound desc = could not find container \"239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c\": container with ID starting with 239fee62961ea5b2d24b67e00930823d9cd220be8a44cbcc4642de1a64afb31c not found: ID does not exist" Jul 10 23:16:30.995802 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:30.995775 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2"] Jul 10 23:16:31.000125 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:31.000103 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728755-hw5k2"] Jul 10 23:16:31.635815 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:16:31.635784 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f191663-9019-40f2-ae81-53508d9acac9" path="/var/lib/kubelet/pods/5f191663-9019-40f2-ae81-53508d9acac9/volumes" Jul 10 23:18:01.704766 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:18:01.704735 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:18:01.709517 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:18:01.709499 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hmgsp_bc260c3c-875f-486d-b49f-2b615b3a4bc0/ovn-acl-logging/0.log" Jul 10 23:20:08.120926 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.120897 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b"] Jul 10 23:20:08.121548 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.121285 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f191663-9019-40f2-ae81-53508d9acac9" containerName="cleanup" Jul 10 23:20:08.121548 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.121299 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f191663-9019-40f2-ae81-53508d9acac9" containerName="cleanup" Jul 10 23:20:08.121548 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.121402 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f191663-9019-40f2-ae81-53508d9acac9" containerName="cleanup" Jul 10 23:20:08.124291 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.124272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.128298 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.128272 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6d6384\"/\"kube-root-ca.crt\"" Jul 10 23:20:08.128417 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.128277 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6d6384\"/\"default-dockercfg-t5vv9\"" Jul 10 23:20:08.128417 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.128369 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6d6384\"/\"openshift-service-ca.crt\"" Jul 10 23:20:08.128417 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.128277 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6d6384\"/\"rate-test-model-6d6384-kserve-self-signed-certs\"" Jul 10 23:20:08.136345 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.136321 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b"] Jul 10 23:20:08.230312 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.230275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-dshm\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.230512 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.230316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ks5\" (UniqueName: \"kubernetes.io/projected/4062bcfe-67bb-41f1-a689-932f170c2418-kube-api-access-66ks5\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.230512 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.230359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-tmp-dir\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.230512 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.230417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-model-cache\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.230512 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.230462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-home\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.230512 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.230482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4062bcfe-67bb-41f1-a689-932f170c2418-tls-certs\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.331486 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.331450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-dshm\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.331658 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.331498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ks5\" (UniqueName: \"kubernetes.io/projected/4062bcfe-67bb-41f1-a689-932f170c2418-kube-api-access-66ks5\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.331658 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.331538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-tmp-dir\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.331658 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.331597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-model-cache\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.331658 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.331623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-home\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.331658 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.331646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4062bcfe-67bb-41f1-a689-932f170c2418-tls-certs\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.332042 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.332018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-model-cache\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.332149 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.332051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-home\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.332207 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.332149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-tmp-dir\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.334286 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.334255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-dshm\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.334612 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.334588 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4062bcfe-67bb-41f1-a689-932f170c2418-tls-certs\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.341053 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.341010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ks5\" (UniqueName: \"kubernetes.io/projected/4062bcfe-67bb-41f1-a689-932f170c2418-kube-api-access-66ks5\") pod \"rate-test-model-6d6384-kserve-74b6df6699-qnc8b\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.433819 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.433784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:08.569209 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:08.569182 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b"] Jul 10 23:20:08.570121 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:20:08.570096 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4062bcfe_67bb_41f1_a689_932f170c2418.slice/crio-3f823c3025397584728de47dd5b097e991022cce92ce4063f632cb75a7e3b89b WatchSource:0}: Error finding container 3f823c3025397584728de47dd5b097e991022cce92ce4063f632cb75a7e3b89b: Status 404 returned error can't find the container with id 3f823c3025397584728de47dd5b097e991022cce92ce4063f632cb75a7e3b89b Jul 10 23:20:09.127573 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:09.127536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" event={"ID":"4062bcfe-67bb-41f1-a689-932f170c2418","Type":"ContainerStarted","Data":"57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d"} Jul 10 23:20:09.127573 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:09.127579 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" event={"ID":"4062bcfe-67bb-41f1-a689-932f170c2418","Type":"ContainerStarted","Data":"3f823c3025397584728de47dd5b097e991022cce92ce4063f632cb75a7e3b89b"} Jul 10 23:20:09.146589 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:09.146530 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" podStartSLOduration=1.146508269 podStartE2EDuration="1.146508269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 23:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:20:09.14375278 +0000 UTC m=+2228.021444417" watchObservedRunningTime="2026-07-10 23:20:09.146508269 +0000 UTC m=+2228.024199891" Jul 10 23:20:18.434060 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:18.434023 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:18.434526 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:18.434075 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:18.450014 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:18.449965 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:20:19.193630 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:20:19.193606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:21:23.972115 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:23.972071 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b"] Jul 10 23:21:38.992871 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:38.992832 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" podUID="4062bcfe-67bb-41f1-a689-932f170c2418" containerName="main" containerID="cri-o://57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d" gracePeriod=45 Jul 10 23:21:39.250793 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.250737 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:21:39.347565 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347538 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-dshm\") pod \"4062bcfe-67bb-41f1-a689-932f170c2418\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " Jul 10 23:21:39.347739 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347626 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-tmp-dir\") pod \"4062bcfe-67bb-41f1-a689-932f170c2418\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " Jul 10 23:21:39.347739 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347650 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-home\") pod \"4062bcfe-67bb-41f1-a689-932f170c2418\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " Jul 10 23:21:39.347739 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347699 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66ks5\" (UniqueName: \"kubernetes.io/projected/4062bcfe-67bb-41f1-a689-932f170c2418-kube-api-access-66ks5\") pod \"4062bcfe-67bb-41f1-a689-932f170c2418\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " Jul 10 23:21:39.347739 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347730 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-model-cache\") pod \"4062bcfe-67bb-41f1-a689-932f170c2418\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " Jul 10 23:21:39.347989 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347754 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4062bcfe-67bb-41f1-a689-932f170c2418-tls-certs\") pod \"4062bcfe-67bb-41f1-a689-932f170c2418\" (UID: \"4062bcfe-67bb-41f1-a689-932f170c2418\") " Jul 10 23:21:39.347989 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347954 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-home" (OuterVolumeSpecName: "home") pod "4062bcfe-67bb-41f1-a689-932f170c2418" (UID: "4062bcfe-67bb-41f1-a689-932f170c2418"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:21:39.347989 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.347955 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4062bcfe-67bb-41f1-a689-932f170c2418" (UID: "4062bcfe-67bb-41f1-a689-932f170c2418"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:21:39.348154 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.348001 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-model-cache" (OuterVolumeSpecName: "model-cache") pod "4062bcfe-67bb-41f1-a689-932f170c2418" (UID: "4062bcfe-67bb-41f1-a689-932f170c2418"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:21:39.348154 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.348050 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-home\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:39.350010 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.349955 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-dshm" (OuterVolumeSpecName: "dshm") pod "4062bcfe-67bb-41f1-a689-932f170c2418" (UID: "4062bcfe-67bb-41f1-a689-932f170c2418"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 23:21:39.350010 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.349960 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4062bcfe-67bb-41f1-a689-932f170c2418-kube-api-access-66ks5" (OuterVolumeSpecName: "kube-api-access-66ks5") pod "4062bcfe-67bb-41f1-a689-932f170c2418" (UID: "4062bcfe-67bb-41f1-a689-932f170c2418"). InnerVolumeSpecName "kube-api-access-66ks5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:21:39.350154 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.350069 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4062bcfe-67bb-41f1-a689-932f170c2418-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4062bcfe-67bb-41f1-a689-932f170c2418" (UID: "4062bcfe-67bb-41f1-a689-932f170c2418"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:21:39.449430 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.449397 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ks5\" (UniqueName: \"kubernetes.io/projected/4062bcfe-67bb-41f1-a689-932f170c2418-kube-api-access-66ks5\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:39.449430 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.449427 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-model-cache\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:39.449430 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.449438 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4062bcfe-67bb-41f1-a689-932f170c2418-tls-certs\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:39.449674 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.449450 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-dshm\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:39.449674 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.449462 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4062bcfe-67bb-41f1-a689-932f170c2418-tmp-dir\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:39.617499 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.615755 2577 generic.go:356] "Generic (PLEG): container finished" podID="4062bcfe-67bb-41f1-a689-932f170c2418" containerID="57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d" exitCode=0 Jul 10 23:21:39.617499 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.615813 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" event={"ID":"4062bcfe-67bb-41f1-a689-932f170c2418","Type":"ContainerDied","Data":"57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d"} Jul 10 23:21:39.617499 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.615852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" event={"ID":"4062bcfe-67bb-41f1-a689-932f170c2418","Type":"ContainerDied","Data":"3f823c3025397584728de47dd5b097e991022cce92ce4063f632cb75a7e3b89b"} Jul 10 23:21:39.617499 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.615884 2577 scope.go:117] "RemoveContainer" containerID="57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d" Jul 10 23:21:39.617499 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.616104 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" Jul 10 23:21:39.626394 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.626377 2577 scope.go:117] "RemoveContainer" containerID="57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d" Jul 10 23:21:39.626650 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:21:39.626630 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d\": container with ID starting with 57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d not found: ID does not exist" containerID="57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d" Jul 10 23:21:39.626701 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.626659 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d"} err="failed to get container status \"57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d\": rpc error: code = NotFound desc = could not find container \"57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d\": container with ID starting with 57a8710fbfc1e07752d88199507abc609347d4374b5d1aea7906a85f1e541a4d not found: ID does not exist" Jul 10 23:21:39.639621 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.639599 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b"] Jul 10 23:21:39.643103 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:39.643081 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b"] Jul 10 23:21:41.635687 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:41.635658 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4062bcfe-67bb-41f1-a689-932f170c2418" path="/var/lib/kubelet/pods/4062bcfe-67bb-41f1-a689-932f170c2418/volumes" Jul 10 23:21:44.184157 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:44.184115 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-6d6384/rate-test-model-6d6384-kserve-74b6df6699-qnc8b" podUID="4062bcfe-67bb-41f1-a689-932f170c2418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Jul 10 23:21:45.547938 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.547905 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c"] Jul 10 23:21:45.548573 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.548537 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" podUID="da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" containerName="maas-api" containerID="cri-o://e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179" gracePeriod=30 Jul 10 23:21:45.797182 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.797159 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:21:45.903750 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.903729 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls\") pod \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " Jul 10 23:21:45.903907 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.903759 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqbp8\" (UniqueName: \"kubernetes.io/projected/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-kube-api-access-cqbp8\") pod \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " Jul 10 23:21:45.903907 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.903783 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-openshift-service-ca\") pod \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\" (UID: \"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444\") " Jul 10 23:21:45.904218 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.904195 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" (UID: "da1b77cb-c4e9-4e22-b0dc-e4ba0585e444"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:21:45.905830 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.905807 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" (UID: "da1b77cb-c4e9-4e22-b0dc-e4ba0585e444"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:21:45.905924 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:45.905865 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-kube-api-access-cqbp8" (OuterVolumeSpecName: "kube-api-access-cqbp8") pod "da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" (UID: "da1b77cb-c4e9-4e22-b0dc-e4ba0585e444"). InnerVolumeSpecName "kube-api-access-cqbp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:21:46.004650 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.004620 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:46.004650 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.004644 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqbp8\" (UniqueName: \"kubernetes.io/projected/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-kube-api-access-cqbp8\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:46.004811 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.004657 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:46.654511 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.654478 2577 generic.go:356] "Generic (PLEG): container finished" podID="da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" containerID="e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179" exitCode=0 Jul 10 23:21:46.654899 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.654522 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" Jul 10 23:21:46.654899 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.654550 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" event={"ID":"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444","Type":"ContainerDied","Data":"e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179"} Jul 10 23:21:46.654899 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.654604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c" event={"ID":"da1b77cb-c4e9-4e22-b0dc-e4ba0585e444","Type":"ContainerDied","Data":"7c87836e1120386b2f3f3f2ef8c786e4e8102c2f5d9d331544ddefdeca8b55b0"} Jul 10 23:21:46.654899 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.654628 2577 scope.go:117] "RemoveContainer" containerID="e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179" Jul 10 23:21:46.664812 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.664794 2577 scope.go:117] "RemoveContainer" containerID="e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179" Jul 10 23:21:46.665077 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:21:46.665056 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179\": container with ID starting with e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179 not found: ID does not exist" containerID="e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179" Jul 10 23:21:46.665173 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.665083 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179"} err="failed to get container status \"e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179\": rpc error: code = NotFound desc = could not find container \"e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179\": container with ID starting with e1166615b69fdf6b4c1614af9eed1c541ca134ed9cb1cbde2da127420ae00179 not found: ID does not exist" Jul 10 23:21:46.678034 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.678013 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c"] Jul 10 23:21:46.682490 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:46.682471 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6d6384-b6d66b798-8d98c"] Jul 10 23:21:47.636013 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:47.635965 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" path="/var/lib/kubelet/pods/da1b77cb-c4e9-4e22-b0dc-e4ba0585e444/volumes" Jul 10 23:21:52.609420 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.609388 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh"] Jul 10 23:21:52.609934 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.609577 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" podUID="06f3b422-a447-40c0-9a28-690805278fd6" containerName="maas-api" containerID="cri-o://9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10" gracePeriod=30 Jul 10 23:21:52.867484 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.867430 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:21:52.966014 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.965956 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwgzj\" (UniqueName: \"kubernetes.io/projected/06f3b422-a447-40c0-9a28-690805278fd6-kube-api-access-lwgzj\") pod \"06f3b422-a447-40c0-9a28-690805278fd6\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " Jul 10 23:21:52.966193 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.966025 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f3b422-a447-40c0-9a28-690805278fd6-openshift-service-ca\") pod \"06f3b422-a447-40c0-9a28-690805278fd6\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " Jul 10 23:21:52.966193 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.966044 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls\") pod \"06f3b422-a447-40c0-9a28-690805278fd6\" (UID: \"06f3b422-a447-40c0-9a28-690805278fd6\") " Jul 10 23:21:52.966396 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.966372 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f3b422-a447-40c0-9a28-690805278fd6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06f3b422-a447-40c0-9a28-690805278fd6" (UID: "06f3b422-a447-40c0-9a28-690805278fd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 23:21:52.968037 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.968014 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06f3b422-a447-40c0-9a28-690805278fd6" (UID: "06f3b422-a447-40c0-9a28-690805278fd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 23:21:52.968157 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:52.968136 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f3b422-a447-40c0-9a28-690805278fd6-kube-api-access-lwgzj" (OuterVolumeSpecName: "kube-api-access-lwgzj") pod "06f3b422-a447-40c0-9a28-690805278fd6" (UID: "06f3b422-a447-40c0-9a28-690805278fd6"). InnerVolumeSpecName "kube-api-access-lwgzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 23:21:53.067435 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.067404 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwgzj\" (UniqueName: \"kubernetes.io/projected/06f3b422-a447-40c0-9a28-690805278fd6-kube-api-access-lwgzj\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:53.067435 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.067431 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f3b422-a447-40c0-9a28-690805278fd6-openshift-service-ca\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:53.067435 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.067441 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f3b422-a447-40c0-9a28-690805278fd6-maas-api-tls\") on node \"ip-10-0-139-196.ec2.internal\" DevicePath \"\"" Jul 10 23:21:53.693288 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.693256 2577 generic.go:356] "Generic (PLEG): container finished" podID="06f3b422-a447-40c0-9a28-690805278fd6" containerID="9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10" exitCode=0 Jul 10 23:21:53.693684 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.693295 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" Jul 10 23:21:53.693684 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.693301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" event={"ID":"06f3b422-a447-40c0-9a28-690805278fd6","Type":"ContainerDied","Data":"9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10"} Jul 10 23:21:53.693684 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.693347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh" event={"ID":"06f3b422-a447-40c0-9a28-690805278fd6","Type":"ContainerDied","Data":"5d2be4b2af95436dde7d80e680081a52fc63daadf09b89c17225d0fab9565985"} Jul 10 23:21:53.693684 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.693369 2577 scope.go:117] "RemoveContainer" containerID="9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10" Jul 10 23:21:53.701746 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.701727 2577 scope.go:117] "RemoveContainer" containerID="9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10" Jul 10 23:21:53.702020 ip-10-0-139-196 kubenswrapper[2577]: E0710 23:21:53.702001 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10\": container with ID starting with 9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10 not found: ID does not exist" containerID="9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10" Jul 10 23:21:53.702092 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.702033 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10"} err="failed to get container status \"9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10\": rpc error: code = NotFound desc = could not find container \"9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10\": container with ID starting with 9324216e4ab7d0e93fee999bb8aa2ce76ced33d6ed339b33d502686698a1ce10 not found: ID does not exist" Jul 10 23:21:53.711405 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.711386 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh"] Jul 10 23:21:53.716281 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:53.716251 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02c498-b7569f98d-bjdbh"] Jul 10 23:21:55.635897 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:21:55.635870 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f3b422-a447-40c0-9a28-690805278fd6" path="/var/lib/kubelet/pods/06f3b422-a447-40c0-9a28-690805278fd6/volumes" Jul 10 23:22:04.207464 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:04.207431 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-547499fdd5-8bvqd_42800ff6-a893-44e8-8064-7cb3cf69267e/manager/0.log" Jul 10 23:22:04.553404 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:04.553324 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-845b6c684-n9pcq_c03c1145-2d83-4a8e-8b3e-9c913005414f/copy-manifests/0.log" Jul 10 23:22:04.572593 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:04.572569 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-845b6c684-n9pcq_c03c1145-2d83-4a8e-8b3e-9c913005414f/manager/0.log" Jul 10 23:22:05.143789 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:05.143758 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-564ccfc6f7-wg9v4_209be334-a47f-4c80-bdee-cc943854a0fa/maas-api/0.log" Jul 10 23:22:05.996957 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:05.996922 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r_0323d7bb-cb80-46c2-bd54-60aa9019cb65/util/0.log" Jul 10 23:22:06.005147 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.005121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r_0323d7bb-cb80-46c2-bd54-60aa9019cb65/pull/0.log" Jul 10 23:22:06.011421 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.011399 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r_0323d7bb-cb80-46c2-bd54-60aa9019cb65/extract/0.log" Jul 10 23:22:06.119204 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.119156 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p_feb3383c-ee94-4856-886f-206be299707b/pull/0.log" Jul 10 23:22:06.125932 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.125907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p_feb3383c-ee94-4856-886f-206be299707b/extract/0.log" Jul 10 23:22:06.135634 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.135617 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p_feb3383c-ee94-4856-886f-206be299707b/util/0.log" Jul 10 23:22:06.242195 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.242176 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv_4792923f-2516-44df-a253-3e2e083ae637/util/0.log" Jul 10 23:22:06.248598 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.248537 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv_4792923f-2516-44df-a253-3e2e083ae637/pull/0.log" Jul 10 23:22:06.254640 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.254622 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv_4792923f-2516-44df-a253-3e2e083ae637/extract/0.log" Jul 10 23:22:06.363887 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.363855 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8_de5ab973-d07a-4701-805c-b6497935a05d/util/0.log" Jul 10 23:22:06.370323 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.370300 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8_de5ab973-d07a-4701-805c-b6497935a05d/pull/0.log" Jul 10 23:22:06.376672 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.376644 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8_de5ab973-d07a-4701-805c-b6497935a05d/extract/0.log" Jul 10 23:22:06.716231 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:06.716192 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fkmbs_030ba25a-8e24-4ad7-ae36-68db15524ea0/manager/0.log" Jul 10 23:22:07.060824 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:07.060747 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-48lm5_e17d4265-b6ec-44b5-b12b-c2018c2ffc2a/manager/0.log" Jul 10 23:22:07.745658 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:07.745619 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-v2g59_60405b33-0de7-4962-aa44-569185c98026/discovery/0.log" Jul 10 23:22:08.740651 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:08.740620 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-q8jpc_94df47bf-7222-4b9f-9994-af2b0d8f3c03/main/0.log" Jul 10 23:22:08.852563 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:08.852538 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-c8mx7_fd049c08-4c89-4a6a-9c65-2d461cb8ead7/main/0.log" Jul 10 23:22:15.668431 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:15.668401 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n2bws_8c4d8088-b537-4bfe-9801-6d6cc21b0939/global-pull-secret-syncer/0.log" Jul 10 23:22:15.801698 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:15.801670 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tg4cx_4edaf4b6-1410-4066-a868-955491cf08fd/konnectivity-agent/0.log" Jul 10 23:22:15.915284 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:15.915257 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-196.ec2.internal_196243ffdd7700b8404231b37ec9c062/haproxy/0.log" Jul 10 23:22:20.015800 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.015762 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r_0323d7bb-cb80-46c2-bd54-60aa9019cb65/extract/0.log" Jul 10 23:22:20.038869 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.038842 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r_0323d7bb-cb80-46c2-bd54-60aa9019cb65/util/0.log" Jul 10 23:22:20.083713 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.083683 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nt55r_0323d7bb-cb80-46c2-bd54-60aa9019cb65/pull/0.log" Jul 10 23:22:20.113288 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.113258 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p_feb3383c-ee94-4856-886f-206be299707b/extract/0.log" Jul 10 23:22:20.144699 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.144671 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p_feb3383c-ee94-4856-886f-206be299707b/util/0.log" Jul 10 23:22:20.169012 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.168960 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07ck6p_feb3383c-ee94-4856-886f-206be299707b/pull/0.log" Jul 10 23:22:20.196528 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.196501 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv_4792923f-2516-44df-a253-3e2e083ae637/extract/0.log" Jul 10 23:22:20.218762 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.218739 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv_4792923f-2516-44df-a253-3e2e083ae637/util/0.log" Jul 10 23:22:20.241326 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.241302 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73k57bv_4792923f-2516-44df-a253-3e2e083ae637/pull/0.log" Jul 10 23:22:20.271420 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.271351 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8_de5ab973-d07a-4701-805c-b6497935a05d/extract/0.log" Jul 10 23:22:20.292696 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.292676 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8_de5ab973-d07a-4701-805c-b6497935a05d/util/0.log" Jul 10 23:22:20.315174 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.315150 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zpfm8_de5ab973-d07a-4701-805c-b6497935a05d/pull/0.log" Jul 10 23:22:20.673034 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.673002 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fkmbs_030ba25a-8e24-4ad7-ae36-68db15524ea0/manager/0.log" Jul 10 23:22:20.930359 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:20.930272 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-48lm5_e17d4265-b6ec-44b5-b12b-c2018c2ffc2a/manager/0.log" Jul 10 23:22:22.929913 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:22.929879 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-87d56d7b-rmrbs_f9d711ea-268c-4d59-bace-1e3e07e8351b/metrics-server/0.log" Jul 10 23:22:22.959815 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:22.959797 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-q4kkd_2291b66c-ba7b-4034-be14-39f408b6d1ef/monitoring-plugin/0.log" Jul 10 23:22:22.993524 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:22.993503 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-khttt_5bbca213-0413-469a-8f42-3ba9c441b435/node-exporter/0.log" Jul 10 23:22:23.020197 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.020174 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-khttt_5bbca213-0413-469a-8f42-3ba9c441b435/kube-rbac-proxy/0.log" Jul 10 23:22:23.052342 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.052322 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-khttt_5bbca213-0413-469a-8f42-3ba9c441b435/init-textfile/0.log" Jul 10 23:22:23.285178 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.285150 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-p4zhs_94ac4c2d-76ab-4d27-8b51-601a33fc8bce/kube-rbac-proxy-main/0.log" Jul 10 23:22:23.311277 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.311253 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-p4zhs_94ac4c2d-76ab-4d27-8b51-601a33fc8bce/kube-rbac-proxy-self/0.log" Jul 10 23:22:23.345619 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.345566 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-p4zhs_94ac4c2d-76ab-4d27-8b51-601a33fc8bce/openshift-state-metrics/0.log" Jul 10 23:22:23.594287 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.594202 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-tjqrv_dab0629d-68f0-46da-ac96-caf194bdfa71/prometheus-operator/0.log" Jul 10 23:22:23.614541 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.614519 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-tjqrv_dab0629d-68f0-46da-ac96-caf194bdfa71/kube-rbac-proxy/0.log" Jul 10 23:22:23.678791 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.678764 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b9b8b755-4v26x_01fd1d17-0881-487a-bf3a-66616fe17363/telemeter-client/0.log" Jul 10 23:22:23.704030 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.704000 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b9b8b755-4v26x_01fd1d17-0881-487a-bf3a-66616fe17363/reload/0.log" Jul 10 23:22:23.734740 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.734717 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b9b8b755-4v26x_01fd1d17-0881-487a-bf3a-66616fe17363/kube-rbac-proxy/0.log" Jul 10 23:22:23.770510 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.770488 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c7d58c74-p4s2w_163c693a-a7a5-4dbc-b8d1-352fca68a8c0/thanos-query/0.log" Jul 10 23:22:23.791806 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.791784 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c7d58c74-p4s2w_163c693a-a7a5-4dbc-b8d1-352fca68a8c0/kube-rbac-proxy-web/0.log" Jul 10 23:22:23.813099 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.813074 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c7d58c74-p4s2w_163c693a-a7a5-4dbc-b8d1-352fca68a8c0/kube-rbac-proxy/0.log" Jul 10 23:22:23.836405 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.836386 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c7d58c74-p4s2w_163c693a-a7a5-4dbc-b8d1-352fca68a8c0/prom-label-proxy/0.log" Jul 10 23:22:23.859774 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.859714 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c7d58c74-p4s2w_163c693a-a7a5-4dbc-b8d1-352fca68a8c0/kube-rbac-proxy-rules/0.log" Jul 10 23:22:23.881118 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:23.881098 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c7d58c74-p4s2w_163c693a-a7a5-4dbc-b8d1-352fca68a8c0/kube-rbac-proxy-metrics/0.log" Jul 10 23:22:26.400776 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.400748 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ff4dd9b5c-qjknf_f044aa54-b6cd-4d3c-998b-095c77297600/console/0.log" Jul 10 23:22:26.440431 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.440404 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9"] Jul 10 23:22:26.440947 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.440925 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" containerName="maas-api" Jul 10 23:22:26.440947 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.440948 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" containerName="maas-api" Jul 10 23:22:26.441188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.440985 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f3b422-a447-40c0-9a28-690805278fd6" containerName="maas-api" Jul 10 23:22:26.441188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.440994 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f3b422-a447-40c0-9a28-690805278fd6" containerName="maas-api" Jul 10 23:22:26.441188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.441014 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4062bcfe-67bb-41f1-a689-932f170c2418" containerName="main" Jul 10 23:22:26.441188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.441023 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4062bcfe-67bb-41f1-a689-932f170c2418" containerName="main" Jul 10 23:22:26.441188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.441148 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4062bcfe-67bb-41f1-a689-932f170c2418" containerName="main" Jul 10 23:22:26.441188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.441162 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f3b422-a447-40c0-9a28-690805278fd6" containerName="maas-api" Jul 10 23:22:26.441188 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.441174 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1b77cb-c4e9-4e22-b0dc-e4ba0585e444" containerName="maas-api" Jul 10 23:22:26.444548 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.444528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.447301 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.447265 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xz2px\"/\"openshift-service-ca.crt\"" Jul 10 23:22:26.448713 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.448688 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xz2px\"/\"kube-root-ca.crt\"" Jul 10 23:22:26.448906 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.448881 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xz2px\"/\"default-dockercfg-shbnr\"" Jul 10 23:22:26.451318 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.451293 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9"] Jul 10 23:22:26.462466 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.462437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-lib-modules\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.462466 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.462465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-podres\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.462654 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.462502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q5v\" (UniqueName: \"kubernetes.io/projected/10a2376f-cf2e-41e8-96c5-1484777544b8-kube-api-access-h6q5v\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.462654 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.462605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-proc\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.462654 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.462637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-sys\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.563644 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-lib-modules\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.563644 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-podres\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.563867 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q5v\" (UniqueName: \"kubernetes.io/projected/10a2376f-cf2e-41e8-96c5-1484777544b8-kube-api-access-h6q5v\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.563909 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-proc\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.563909 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-podres\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.564011 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-sys\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.564011 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-proc\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.564011 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.563778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-lib-modules\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.564126 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.564053 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10a2376f-cf2e-41e8-96c5-1484777544b8-sys\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.571920 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.571900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q5v\" (UniqueName: \"kubernetes.io/projected/10a2376f-cf2e-41e8-96c5-1484777544b8-kube-api-access-h6q5v\") pod \"perf-node-gather-daemonset-gjpx9\" (UID: \"10a2376f-cf2e-41e8-96c5-1484777544b8\") " pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.757246 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.757144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:26.890217 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:26.890196 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9"] Jul 10 23:22:26.891385 ip-10-0-139-196 kubenswrapper[2577]: W0710 23:22:26.891357 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod10a2376f_cf2e_41e8_96c5_1484777544b8.slice/crio-a1ad8c032362435b6cf0898da3169373df45134e962fdd302f8d2f9a6d5371ec WatchSource:0}: Error finding container a1ad8c032362435b6cf0898da3169373df45134e962fdd302f8d2f9a6d5371ec: Status 404 returned error can't find the container with id a1ad8c032362435b6cf0898da3169373df45134e962fdd302f8d2f9a6d5371ec Jul 10 23:22:27.871056 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:27.871018 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" event={"ID":"10a2376f-cf2e-41e8-96c5-1484777544b8","Type":"ContainerStarted","Data":"ef51b9928190e87c99db7a2504ad3ae48bc3305e5a994171806e01e018a5353a"} Jul 10 23:22:27.871431 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:27.871083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" event={"ID":"10a2376f-cf2e-41e8-96c5-1484777544b8","Type":"ContainerStarted","Data":"a1ad8c032362435b6cf0898da3169373df45134e962fdd302f8d2f9a6d5371ec"} Jul 10 23:22:27.871431 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:27.871111 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:27.888114 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:27.888064 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" podStartSLOduration=1.888047926 podStartE2EDuration="1.888047926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:22:27.886544946 +0000 UTC m=+2366.764236569" watchObservedRunningTime="2026-07-10 23:22:27.888047926 +0000 UTC m=+2366.765739590" Jul 10 23:22:28.375600 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:28.375573 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqks5_3ad68902-2c53-433c-a64b-561a56cee8d4/dns/0.log" Jul 10 23:22:28.397489 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:28.397460 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqks5_3ad68902-2c53-433c-a64b-561a56cee8d4/kube-rbac-proxy/0.log" Jul 10 23:22:28.421122 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:28.421090 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ffv7g_578887bd-bedf-4ef4-ad56-c7585e1573cb/dns-node-resolver/0.log" Jul 10 23:22:29.054295 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:29.054268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5f5gr_a046f2e1-b229-43a0-b693-1609f19e59dc/node-ca/0.log" Jul 10 23:22:30.223568 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:30.223530 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-v2g59_60405b33-0de7-4962-aa44-569185c98026/discovery/0.log" Jul 10 23:22:31.127134 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:31.127106 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-65vzt_f6e50fe6-8a37-4770-9f2a-377db6af6cba/serve-healthcheck-canary/0.log" Jul 10 23:22:31.954693 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:31.954665 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spnht_f1988cf5-429e-4b86-8a35-0ee1ddabd8e9/kube-rbac-proxy/0.log" Jul 10 23:22:31.975768 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:31.975747 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spnht_f1988cf5-429e-4b86-8a35-0ee1ddabd8e9/exporter/0.log" Jul 10 23:22:31.997182 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:31.997154 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-spnht_f1988cf5-429e-4b86-8a35-0ee1ddabd8e9/extractor/0.log" Jul 10 23:22:33.886168 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:33.886142 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xz2px/perf-node-gather-daemonset-gjpx9" Jul 10 23:22:34.418811 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:34.418762 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-547499fdd5-8bvqd_42800ff6-a893-44e8-8064-7cb3cf69267e/manager/0.log" Jul 10 23:22:34.613739 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:34.613708 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-845b6c684-n9pcq_c03c1145-2d83-4a8e-8b3e-9c913005414f/manager/0.log" Jul 10 23:22:34.631682 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:34.631662 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-845b6c684-n9pcq_c03c1145-2d83-4a8e-8b3e-9c913005414f/copy-manifests/0.log" Jul 10 23:22:36.133660 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:36.133579 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c7f744b8d-xlkc4_e5549f83-0ca9-4147-a4f9-e45b5fab53fb/manager/0.log" Jul 10 23:22:36.172296 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:36.172262 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2dsgc_ab44585a-c7d4-4a1b-81cd-bdd7b0b2c826/openshift-lws-operator/0.log" Jul 10 23:22:37.104179 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:37.104155 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-v2g59_60405b33-0de7-4962-aa44-569185c98026/discovery/0.log" Jul 10 23:22:38.658433 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:38.658399 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-v2g59_60405b33-0de7-4962-aa44-569185c98026/discovery/0.log" Jul 10 23:22:43.150887 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.150857 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4hfp_5c654044-de95-4939-89e1-92b2618e6e56/kube-multus-additional-cni-plugins/0.log" Jul 10 23:22:43.177435 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.177411 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4hfp_5c654044-de95-4939-89e1-92b2618e6e56/egress-router-binary-copy/0.log" Jul 10 23:22:43.207051 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.207024 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4hfp_5c654044-de95-4939-89e1-92b2618e6e56/cni-plugins/0.log" Jul 10 23:22:43.234042 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.234020 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4hfp_5c654044-de95-4939-89e1-92b2618e6e56/bond-cni-plugin/0.log" Jul 10 23:22:43.260076 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.260050 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4hfp_5c654044-de95-4939-89e1-92b2618e6e56/routeoverride-cni/0.log" Jul 10 23:22:43.286729 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.286708 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4hfp_5c654044-de95-4939-89e1-92b2618e6e56/whereabouts-cni-bincopy/0.log" Jul 10 23:22:43.313523 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.313498 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4hfp_5c654044-de95-4939-89e1-92b2618e6e56/whereabouts-cni/0.log" Jul 10 23:22:43.421882 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.421793 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gb4pk_bd4ef950-777f-4f6a-9bd4-d08710af09b8/kube-multus/0.log" Jul 10 23:22:43.558002 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.557961 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9hnxs_c79360d2-84bf-4ebf-b7bc-e1e818e43cb0/network-metrics-daemon/0.log" Jul 10 23:22:43.579643 ip-10-0-139-196 kubenswrapper[2577]: I0710 23:22:43.579617 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9hnxs_c79360d2-84bf-4ebf-b7bc-e1e818e43cb0/kube-rbac-proxy/0.log"