Jul 09 14:13:53.176531 ip-10-0-141-45 systemd[1]: Starting Kubernetes Kubelet... Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.635603 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643437 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643450 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643454 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643457 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643460 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643464 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643467 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:13:53.661555 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643469 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643472 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643475 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643479 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643481 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643484 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643487 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643490 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643493 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643496 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643498 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643501 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643504 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643507 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643509 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643514 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643517 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643520 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643524 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:13:53.663566 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643526 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643529 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643532 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643534 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643537 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643540 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643543 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643545 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643548 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643550 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643553 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643555 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643558 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643560 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643563 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643566 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643569 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643571 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643574 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643577 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:13:53.665865 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643579 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643582 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643585 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643587 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643591 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643593 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643596 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643598 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643601 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643604 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643606 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643609 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643611 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643614 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643616 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643619 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643621 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643624 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643626 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643629 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:13:53.667860 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643631 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643634 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643636 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643638 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643641 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643644 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643647 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643650 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643653 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643655 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643658 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643663 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643666 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643669 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643674 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643677 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643682 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643686 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643689 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:13:53.670056 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643691 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643695 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643698 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643701 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643704 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643706 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643709 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643711 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643714 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643717 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643719 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643722 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643724 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.643727 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644471 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644478 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644481 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644483 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644486 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:13:53.672045 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644489 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644494 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644498 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644502 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644505 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644508 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644511 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644514 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644517 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644521 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644524 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644528 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644531 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644533 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644536 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644539 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644543 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644547 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644550 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:13:53.674334 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644553 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644556 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644558 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644561 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644564 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644566 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644569 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644572 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644575 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644577 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644580 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644582 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644585 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644587 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644595 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644598 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644601 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644606 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644609 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644611 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:13:53.676092 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644614 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644616 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644619 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644622 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644624 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644627 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644630 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644632 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644635 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644637 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644640 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644642 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644645 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644648 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644651 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644654 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644657 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644659 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644662 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644665 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:13:53.678419 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644668 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644671 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644674 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644676 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644680 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644684 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644687 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644696 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644699 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644701 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644704 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644707 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644709 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644712 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644715 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644717 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644720 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644722 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644725 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644728 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:13:53.680254 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644730 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644733 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644735 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644738 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644740 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644743 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644745 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644748 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644751 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644754 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644756 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644758 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644761 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644763 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.644766 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645181 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645187 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645190 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645192 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:13:53.682655 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645196 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645198 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645201 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645204 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645206 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645209 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645212 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645215 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645217 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645233 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645236 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645239 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645242 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645245 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645247 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645250 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645254 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645258 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645262 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:13:53.684729 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645266 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645269 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645272 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645275 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645278 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645280 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645283 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645285 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645288 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645290 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645293 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645296 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645298 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645301 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645304 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645307 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645310 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645313 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645315 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645318 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:13:53.686858 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645321 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645323 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645326 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645331 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645335 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645341 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645344 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645347 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645352 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645355 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645358 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645361 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645364 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645367 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645370 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645373 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645376 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645379 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645382 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645385 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:13:53.688602 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645388 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645390 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645393 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645395 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645398 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645401 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645403 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645406 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645409 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645411 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645414 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645416 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645419 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645421 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645424 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645427 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645429 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645433 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645436 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645438 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:13:53.691032 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645443 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645446 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645448 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645451 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645453 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645456 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645458 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645461 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645463 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645466 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645469 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645471 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645474 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645476 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645478 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.645481 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645552 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645560 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645566 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645571 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 14:13:53.692820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645576 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645579 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645583 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645588 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645591 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645593 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645597 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645600 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645603 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645607 2575 flags.go:64] FLAG: --cgroup-root="" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645610 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645613 2575 flags.go:64] FLAG: --client-ca-file="" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645618 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645621 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645625 2575 flags.go:64] FLAG: --cluster-domain="" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645628 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645631 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645634 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645638 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645641 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645644 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645647 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645651 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645654 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 09 14:13:53.695004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645657 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645659 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645663 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645665 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645669 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645672 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645675 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645678 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645681 2575 flags.go:64] FLAG: --enable-server="true" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645685 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645695 2575 flags.go:64] FLAG: --event-burst="100" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645698 2575 flags.go:64] FLAG: --event-qps="50" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645701 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645704 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645707 2575 flags.go:64] FLAG: --eviction-hard="" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645710 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645713 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645718 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645721 2575 flags.go:64] FLAG: --eviction-soft="" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645724 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645728 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645731 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645734 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645737 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645740 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 14:13:53.696879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645743 2575 flags.go:64] FLAG: --feature-gates="" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645747 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645750 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645753 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645756 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645759 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645762 2575 flags.go:64] FLAG: --help="false" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645765 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-141-45.ec2.internal" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645768 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645771 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645774 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645777 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645781 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645784 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645787 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645789 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645792 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645795 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645798 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645801 2575 flags.go:64] FLAG: --kube-reserved="" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645805 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645807 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645810 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645813 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 14:13:53.699478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645817 2575 flags.go:64] FLAG: --lock-file="" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645820 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645823 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645827 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645833 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645836 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645838 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645841 2575 flags.go:64] FLAG: --logging-format="text" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645844 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645847 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645850 2575 flags.go:64] FLAG: --manifest-url="" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645853 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645857 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645860 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645864 2575 flags.go:64] FLAG: --max-pods="110" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645867 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645870 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645873 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645876 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645879 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645882 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645885 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645893 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645896 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645898 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 14:13:53.713858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645901 2575 flags.go:64] FLAG: --pod-cidr="" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645905 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645911 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645914 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645917 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645920 2575 flags.go:64] FLAG: --port="10250" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645923 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645926 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08dae14de0498e769" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645930 2575 flags.go:64] FLAG: --qos-reserved="" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645933 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645937 2575 flags.go:64] FLAG: --register-node="true" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645940 2575 flags.go:64] FLAG: --register-with-taints="" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645944 2575 flags.go:64] FLAG: --registry-burst="10" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645947 2575 flags.go:64] FLAG: --registry-qps="5" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645949 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645952 2575 flags.go:64] FLAG: --reserved-memory="" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645955 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645959 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645962 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645964 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645967 2575 flags.go:64] FLAG: --runonce="false" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645970 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645973 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645976 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645979 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 14:13:53.723427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645982 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 14:13:53.722809 ip-10-0-141-45 systemd[1]: Started Kubernetes Kubelet. Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645984 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645988 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645991 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645994 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.645997 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646000 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646002 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646005 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646009 2575 flags.go:64] FLAG: --system-cgroups="" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646012 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646017 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646020 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646023 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646027 2575 flags.go:64] FLAG: --tls-min-version="" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646031 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646034 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646039 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646042 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646045 2575 flags.go:64] FLAG: --v="2" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646049 2575 flags.go:64] FLAG: --version="false" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646053 2575 flags.go:64] FLAG: --vmodule="" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646057 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.646060 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646203 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:13:53.932564 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646207 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646211 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646215 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646233 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646236 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646239 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646242 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646244 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646247 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646250 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646253 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646256 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646258 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646261 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646264 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646268 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646271 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646275 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646278 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:13:53.956493 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646281 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646283 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646286 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646291 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646293 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646297 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646300 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646303 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646305 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646308 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646311 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646313 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646316 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646320 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646323 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646326 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646329 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646332 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646335 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646337 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:13:53.958601 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646340 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646342 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646345 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646348 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646350 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646353 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646355 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646358 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646360 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646363 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646365 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646368 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646371 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646373 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646376 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646380 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646383 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646387 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646389 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646392 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:13:53.959801 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646394 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646397 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646400 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646402 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646405 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646408 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646410 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646413 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646415 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646418 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646420 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646423 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646425 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646428 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646431 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646433 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646436 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646439 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646442 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646444 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:13:53.960856 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646447 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646449 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646452 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646454 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646457 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646460 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646462 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646466 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646469 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646472 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646475 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646477 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646480 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646483 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646485 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646488 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646490 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646493 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:13:53.961971 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.646495 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.647201 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.654364 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.654381 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.655323 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.655331 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655389 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655395 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655398 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655403 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655406 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655408 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655412 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655415 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655418 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:13:53.962850 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655421 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655424 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655427 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655430 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655433 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655436 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655439 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655442 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655445 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655447 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655450 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655453 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655456 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655458 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655461 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655464 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655467 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655471 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655475 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655478 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:13:53.963722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655482 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655485 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655488 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655491 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655494 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655497 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655499 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655502 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655504 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655507 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655509 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655512 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655514 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655517 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655521 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655525 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655528 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655531 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:13:53.964639 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655536 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655539 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655543 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655546 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655548 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655551 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655554 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655556 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655559 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655561 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655564 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655567 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655570 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655573 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655575 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655578 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655581 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655584 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655586 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655589 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:13:53.965552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655591 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655594 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655596 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655599 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655601 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655604 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655607 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655609 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655612 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655614 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655617 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655619 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655623 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655626 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655628 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655630 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655633 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655636 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655643 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655646 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:13:53.966635 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655649 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655652 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655654 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655657 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655659 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655662 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655665 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655667 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655670 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655673 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655675 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655678 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.655683 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655817 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655823 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:13:53.967722 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655826 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655829 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655832 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655835 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655838 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655841 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655843 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655846 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655848 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655852 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655855 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655859 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655862 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655865 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655868 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655871 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655875 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655877 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655880 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655882 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:13:53.968528 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655885 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655887 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655890 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655893 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655895 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655898 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655904 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655907 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655909 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655912 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655914 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655917 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655919 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655922 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655924 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655927 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655929 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655932 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655934 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655937 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:13:53.969627 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655939 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655943 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655946 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655948 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655951 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655953 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655956 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655958 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655962 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655965 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655967 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655970 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655972 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655975 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655978 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655980 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655983 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655985 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655987 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655990 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:13:53.970499 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655993 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655996 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.655998 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656001 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656003 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656006 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656008 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656011 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656013 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656016 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656019 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656023 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656026 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656030 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656033 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656035 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656038 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656041 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656043 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:13:53.971588 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656046 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656049 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656052 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656055 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656058 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656060 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656063 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656067 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656070 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656073 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656075 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656078 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656080 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656083 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656086 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656088 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656091 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:13:53.972469 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:53.656093 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.656097 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.656211 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.658692 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.660635 2575 server.go:1019] "Starting client certificate rotation" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.660743 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.660786 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.666497 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.670736 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.682579 2575 log.go:25] "Validated CRI v1 runtime API" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.686037 2575 log.go:25] "Validated CRI v1 image API" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.689009 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 14:13:53.973605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.692978 2575 fs.go:135] Filesystem UUIDs: map[22969c72-41ad-4c2e-9a4f-e2d3359bc0a6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a1cc6001-187d-4fa5-aff4-2d593bf02e08:/dev/nvme0n1p3] Jul 09 14:13:53.974467 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.693000 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 14:13:53.974680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.697272 2575 manager.go:216] Machine: {Timestamp:2026-07-09 14:13:53.696782481 +0000 UTC m=+0.385711731 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100337 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2850bf5972cc06b00328272766992e SystemUUID:ec2850bf-5972-cc06-b003-28272766992e BootID:145c069b-27ee-4b5a-91c8-cb05c7db480a Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:74:de:32:e9:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:74:de:32:e9:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:44:d2:98:ee:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 14:13:53.974680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.697369 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 14:13:53.974680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.697442 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 14:13:53.974680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.699372 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 14:13:53.974680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.701216 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 14:13:53.974680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.701256 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 14:13:53.974680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.702162 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.702171 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.702178 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.702192 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.703631 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.708521 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.708649 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.708678 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.708693 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.708705 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.708716 2575 kubelet.go:398] "Adding apiserver pod source" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.708725 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.709106 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.709121 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.711821 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.714405 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.714434 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715632 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715647 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715653 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715659 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715667 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715675 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715686 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715693 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715699 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715705 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715713 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.715718 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.716709 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.716720 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.720041 2575 server.go:1268] "Started kubelet" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.721207 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.721542 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.721657 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.723212 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.723649 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.723667 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.723777 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 14:13:53.977259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.724705 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.731340 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.732097 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.732123 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.732417 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.732966 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.732979 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.732985 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733118 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733127 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733338 2575 factory.go:55] Registering systemd factory Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733353 2575 factory.go:223] Registration of the systemd container factory successfully Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733588 2575 factory.go:153] Registering CRI-O factory Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733603 2575 factory.go:223] Registration of the crio container factory successfully Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733655 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733675 2575 factory.go:103] Registering Raw factory Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.733689 2575 manager.go:1196] Started watching for new ooms in manager Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.734067 2575 manager.go:349] Starting recovery of all containers Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.735894 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.735962 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-45.ec2.internal.18c0a499fcfb7dd1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-45.ec2.internal,UID:ip-10-0-141-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-45.ec2.internal,},FirstTimestamp:2026-07-09 14:13:53.720008145 +0000 UTC m=+0.408937374,LastTimestamp:2026-07-09 14:13:53.720008145 +0000 UTC m=+0.408937374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-45.ec2.internal,}" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.738247 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.745397 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.746699 2575 manager.go:354] Recovery completed Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.752108 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.754475 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.754511 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.754523 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.754973 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.754982 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.754999 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.755032 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.756280 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.756303 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:53.979012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.756313 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.757358 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-45.ec2.internal.18c0a499ff09c0d5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-45.ec2.internal,UID:ip-10-0-141-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-45.ec2.internal,},FirstTimestamp:2026-07-09 14:13:53.754497237 +0000 UTC m=+0.443426467,LastTimestamp:2026-07-09 14:13:53.754497237 +0000 UTC m=+0.443426467,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-45.ec2.internal,}" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.759873 2575 policy_none.go:49] "None policy: Start" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.759890 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.759903 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.760468 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kfdql" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.761903 2575 policy_none.go:47] "Start" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.767067 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kfdql" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.768347 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-45.ec2.internal.18c0a499ff0a0aad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-45.ec2.internal,UID:ip-10-0-141-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-45.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-45.ec2.internal,},FirstTimestamp:2026-07-09 14:13:53.754516141 +0000 UTC m=+0.443445371,LastTimestamp:2026-07-09 14:13:53.754516141 +0000 UTC m=+0.443445371,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-45.ec2.internal,}" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.770870 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-45.ec2.internal.18c0a499ff0a3f95 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-45.ec2.internal,UID:ip-10-0-141-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-45.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-45.ec2.internal,},FirstTimestamp:2026-07-09 14:13:53.754529685 +0000 UTC m=+0.443458917,LastTimestamp:2026-07-09 14:13:53.754529685 +0000 UTC m=+0.443458917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-45.ec2.internal,}" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.795926 2575 manager.go:337] "Starting Device Plugin manager" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.795961 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.795974 2575 server.go:87] "Starting device plugin registration server" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.796242 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.796254 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.796346 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.796431 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.796438 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.797515 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.797555 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.879408 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 14:13:53.980962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.879445 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.879466 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.879498 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.882374 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.896822 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.897926 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.897958 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.897972 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.897994 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-45.ec2.internal" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.911275 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-45.ec2.internal" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.911294 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-45.ec2.internal\": node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:53.943035 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.980609 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"] Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.980702 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.981753 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.981782 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:53.982065 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.981795 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:53.987671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.984338 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:53.987671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.984434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:53.987671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.984461 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.992237 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.992271 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.992273 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.992286 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.992299 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.992312 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.994624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.994653 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.995342 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.995371 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:53.995385 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.017963 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.023792 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.034740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.034760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.034776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1c7bc3b4c133f1b0578384be47f8d38-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"a1c7bc3b4c133f1b0578384be47f8d38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.114756 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.043835 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.135427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1c7bc3b4c133f1b0578384be47f8d38-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"a1c7bc3b4c133f1b0578384be47f8d38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.135451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1c7bc3b4c133f1b0578384be47f8d38-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"a1c7bc3b4c133f1b0578384be47f8d38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.135501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.135516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.135553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.135583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34b43d5f526b7974ac738b5aaab0d959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"34b43d5f526b7974ac738b5aaab0d959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.144495 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.257675 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.245319 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.365266 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.320533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.365266 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.326707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 09 14:13:54.365266 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.345606 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.476912 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.446208 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.587916 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.546708 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.712820 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.647556 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.712820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.659757 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 14:13:54.712820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.659919 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 14:13:54.830402 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.748525 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.830402 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.773422 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 14:08:54 +0000 UTC" deadline="2028-04-07 18:58:57.774739722 +0000 UTC" Jul 09 14:13:54.830402 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.773463 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15316h45m3.001281861s" Jul 09 14:13:54.946314 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.849423 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:54.946314 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.850511 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:13:54.946314 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:54.907240 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:13:55.051396 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:54.950413 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:55.051396 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.051117 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found" Jul 09 14:13:55.071241 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.071216 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:13:55.133768 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.133678 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" Jul 09 14:13:55.140435 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.140408 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:13:55.140542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.140453 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:13:55.140542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.140468 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:13:55.145929 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.145909 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 14:13:55.147600 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.147583 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" Jul 09 14:13:55.160389 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.160369 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 14:13:55.218432 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.218398 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:13:55.476881 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.476797 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:13:55.710380 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.710351 2575 apiserver.go:52] "Watching apiserver" Jul 09 14:13:55.718076 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.718049 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 14:13:55.719385 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.719359 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-d22n5","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal","openshift-image-registry/node-ca-9cbwm","openshift-multus/multus-additional-cni-plugins-mwrvj","openshift-network-diagnostics/network-check-target-lfhcd","openshift-ovn-kubernetes/ovnkube-node-gw9lp","kube-system/global-pull-secret-syncer-bgmqg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx","openshift-cluster-node-tuning-operator/tuned-tc9w9","openshift-dns/node-resolver-9fdr6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal","openshift-multus/multus-8954f","openshift-multus/network-metrics-daemon-8jdld","openshift-network-operator/iptables-alerter-sz4qc"] Jul 09 14:13:55.722330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.722310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:55.724810 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.724790 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.725015 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.724927 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 14:13:55.725134 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.725031 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6ttcw\"" Jul 09 14:13:55.725134 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.725038 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 14:13:55.727253 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.727196 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 14:13:55.727253 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.727220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zl7cb\"" Jul 09 14:13:55.727253 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.727263 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 14:13:55.727482 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.727408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.729647 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.729628 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 14:13:55.729834 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.729810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.729955 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.729848 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 14:13:55.729955 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.729872 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 14:13:55.729955 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.729900 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 14:13:55.729955 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.729940 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 14:13:55.730214 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.730196 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n5s5t\"" Jul 09 14:13:55.732386 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.732366 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 14:13:55.732523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.732486 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:55.732617 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.732592 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:13:55.732836 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.732814 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 14:13:55.732938 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.732868 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 14:13:55.733107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.733088 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zzl52\"" Jul 09 14:13:55.733160 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.733105 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 14:13:55.734882 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.734863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.737247 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.737226 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 14:13:55.737418 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.737273 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 14:13:55.737562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.737544 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.737602 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.737583 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 14:13:55.739970 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.737860 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:13:55.739970 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.739582 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 14:13:55.739970 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.739655 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 14:13:55.739970 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.739767 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 14:13:55.739970 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.739883 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5krs4\"" Jul 09 14:13:55.742611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.742585 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.744938 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.744920 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.745084 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.745062 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 14:13:55.745177 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.745094 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 14:13:55.745475 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.745456 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 14:13:55.746471 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-run-ovn-kubernetes\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746581 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742z9\" (UniqueName: \"kubernetes.io/projected/2b62be6d-4dc8-4319-8e0a-29e78e268b31-kube-api-access-742z9\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746581 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746491 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fm4md\"" Jul 09 14:13:55.746581 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0115aff2-b98a-4415-8204-4de41320eb9a-serviceca\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.746581 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-slash\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746581 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-run-netns\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-ovn\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad276ab-66e3-4cc8-a0a5-6749b625419b-hosts-file\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hf6p\" (UniqueName: \"kubernetes.io/projected/aad276ab-66e3-4cc8-a0a5-6749b625419b-kube-api-access-4hf6p\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-kubelet\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-systemd-units\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovnkube-config\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.746838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovn-node-metrics-cert\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0115aff2-b98a-4415-8204-4de41320eb9a-host\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dc9\" (UniqueName: \"kubernetes.io/projected/0115aff2-b98a-4415-8204-4de41320eb9a-kube-api-access-27dc9\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-var-lib-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-node-log\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-log-socket\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.746982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-env-overrides\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad276ab-66e3-4cc8-a0a5-6749b625419b-tmp-dir\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cnibin\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovnkube-script-lib\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-os-release\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747230 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-cni-netd\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747196 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-system-cni-dir\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-systemd\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-etc-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86ae436a-86e2-4812-86cd-7ca9b4adc1b5-agent-certs\") pod \"konnectivity-agent-d22n5\" (UID: \"86ae436a-86e2-4812-86cd-7ca9b4adc1b5\") " pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86ae436a-86e2-4812-86cd-7ca9b4adc1b5-konnectivity-ca\") pod \"konnectivity-agent-d22n5\" (UID: \"86ae436a-86e2-4812-86cd-7ca9b4adc1b5\") " pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bglsw\" (UniqueName: \"kubernetes.io/projected/2323e31a-ed38-40a0-a730-ef6ef0acc26a-kube-api-access-bglsw\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-cni-bin\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8954f" Jul 09 14:13:55.747918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747867 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:13:55.748686 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.747997 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhhjg\"" Jul 09 14:13:55.748686 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.748105 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 14:13:55.749903 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.749887 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tfwcm\"" Jul 09 14:13:55.750202 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.750170 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:55.750325 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.750173 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 14:13:55.750325 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.750264 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:13:55.750325 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.750290 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 14:13:55.752778 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.752760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.754933 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.754918 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:13:55.755411 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.755387 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 14:13:55.755567 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.755546 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jxfsv\"" Jul 09 14:13:55.755644 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.755605 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 14:13:55.775596 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.775575 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vwtqj" Jul 09 14:13:55.783217 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.783092 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vwtqj" Jul 09 14:13:55.822640 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:55.822602 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b43d5f526b7974ac738b5aaab0d959.slice/crio-11f6197a9c6d60dc13233f62a18034112c44099b1bbd0a43ab6f56c79bedf439 WatchSource:0}: Error finding container 11f6197a9c6d60dc13233f62a18034112c44099b1bbd0a43ab6f56c79bedf439: Status 404 returned error can't find the container with id 11f6197a9c6d60dc13233f62a18034112c44099b1bbd0a43ab6f56c79bedf439 Jul 09 14:13:55.828726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.828706 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:13:55.833745 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.833726 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 14:13:55.848043 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-tmp\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.848043 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-os-release\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.848212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0115aff2-b98a-4415-8204-4de41320eb9a-host\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.848212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dc9\" (UniqueName: \"kubernetes.io/projected/0115aff2-b98a-4415-8204-4de41320eb9a-kube-api-access-27dc9\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.848212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-var-lib-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848156 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0115aff2-b98a-4415-8204-4de41320eb9a-host\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.848212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-log-socket\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-env-overrides\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-var-lib-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b950518d-63fa-4cdc-92af-2f6218d8da67-iptables-alerter-script\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-log-socket\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad276ab-66e3-4cc8-a0a5-6749b625419b-tmp-dir\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b950518d-63fa-4cdc-92af-2f6218d8da67-host-slash\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-cni-netd\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-device-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848465 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-cni-netd\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-modprobe-d\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.848534 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848533 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysctl-conf\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848562 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-tuned\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysconfig\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad276ab-66e3-4cc8-a0a5-6749b625419b-tmp-dir\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s22nx\" (UniqueName: \"kubernetes.io/projected/e810efc5-e567-4924-af70-18dd39817937-kube-api-access-s22nx\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-system-cni-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5effc12-cbb1-4745-a99b-9fdab63138af-cni-binary-copy\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-cni-bin\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-daemon-config\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-env-overrides\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-multus-certs\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86ae436a-86e2-4812-86cd-7ca9b4adc1b5-konnectivity-ca\") pod \"konnectivity-agent-d22n5\" (UID: \"86ae436a-86e2-4812-86cd-7ca9b4adc1b5\") " pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bglsw\" (UniqueName: \"kubernetes.io/projected/2323e31a-ed38-40a0-a730-ef6ef0acc26a-kube-api-access-bglsw\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-cni-bin\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849226 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-run-ovn-kubernetes\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-cni-bin\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742z9\" (UniqueName: \"kubernetes.io/projected/2b62be6d-4dc8-4319-8e0a-29e78e268b31-kube-api-access-742z9\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.848982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-sys-fs\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-run-ovn-kubernetes\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-cnibin\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-slash\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-slash\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-run-netns\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-ovn\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwmqz\" (UniqueName: \"kubernetes.io/projected/1168fa75-355f-49ac-aabf-50aa7913d802-kube-api-access-cwmqz\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-run-netns\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.849914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2cdcb585-98f7-4c44-928e-68b3dc41164e-kubelet-config\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-ovn\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-socket-dir-parent\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-k8s-cni-cncf-io\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-netns\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad276ab-66e3-4cc8-a0a5-6749b625419b-hosts-file\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hf6p\" (UniqueName: \"kubernetes.io/projected/aad276ab-66e3-4cc8-a0a5-6749b625419b-kube-api-access-4hf6p\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-systemd-units\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad276ab-66e3-4cc8-a0a5-6749b625419b-hosts-file\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-systemd-units\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovnkube-config\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86ae436a-86e2-4812-86cd-7ca9b4adc1b5-konnectivity-ca\") pod \"konnectivity-agent-d22n5\" (UID: \"86ae436a-86e2-4812-86cd-7ca9b4adc1b5\") " pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-systemd\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.850611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-kube-api-access-55z7n\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-cni-multus\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-node-log\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysctl-d\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-etc-kubernetes\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849600 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-node-log\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cnibin\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cnibin\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovnkube-script-lib\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-sys\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-var-lib-kubelet\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.851072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtzn\" (UniqueName: \"kubernetes.io/projected/e5effc12-cbb1-4745-a99b-9fdab63138af-kube-api-access-jmtzn\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-os-release\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsssq\" (UniqueName: \"kubernetes.io/projected/b950518d-63fa-4cdc-92af-2f6218d8da67-kube-api-access-fsssq\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-os-release\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849964 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovnkube-config\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.849991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-system-cni-dir\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-systemd\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-etc-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-socket-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850107 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-run-systemd\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850120 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-registration-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.851523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-etc-openvswitch\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850071 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-system-cni-dir\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-kubelet\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovnkube-script-lib\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-hostroot\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86ae436a-86e2-4812-86cd-7ca9b4adc1b5-agent-certs\") pod \"konnectivity-agent-d22n5\" (UID: \"86ae436a-86e2-4812-86cd-7ca9b4adc1b5\") " pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-run\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-lib-modules\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-cni-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-conf-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850441 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2cdcb585-98f7-4c44-928e-68b3dc41164e-dbus\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0115aff2-b98a-4415-8204-4de41320eb9a-serviceca\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2323e31a-ed38-40a0-a730-ef6ef0acc26a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.851920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850547 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2323e31a-ed38-40a0-a730-ef6ef0acc26a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-kubelet\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850577 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b62be6d-4dc8-4319-8e0a-29e78e268b31-host-kubelet\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovn-node-metrics-cert\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-etc-selinux\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-kubernetes\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.850784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-host\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.852356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.851547 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0115aff2-b98a-4415-8204-4de41320eb9a-serviceca\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.853843 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.853825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b62be6d-4dc8-4319-8e0a-29e78e268b31-ovn-node-metrics-cert\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.854003 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.853947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86ae436a-86e2-4812-86cd-7ca9b4adc1b5-agent-certs\") pod \"konnectivity-agent-d22n5\" (UID: \"86ae436a-86e2-4812-86cd-7ca9b4adc1b5\") " pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:55.861840 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.861815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dc9\" (UniqueName: \"kubernetes.io/projected/0115aff2-b98a-4415-8204-4de41320eb9a-kube-api-access-27dc9\") pod \"node-ca-9cbwm\" (UID: \"0115aff2-b98a-4415-8204-4de41320eb9a\") " pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:55.861958 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.861912 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:13:55.861958 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.861935 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:13:55.861958 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.861951 2575 projected.go:196] Error preparing data for projected volume kube-api-access-r9tmd for pod openshift-network-diagnostics/network-check-target-lfhcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:55.862062 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.862042 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd podName:50732da1-a962-4c24-8c2e-f71c931e04c1 nodeName:}" failed. No retries permitted until 2026-07-09 14:13:56.362017917 +0000 UTC m=+3.050947140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r9tmd" (UniqueName: "kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd") pod "network-check-target-lfhcd" (UID: "50732da1-a962-4c24-8c2e-f71c931e04c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:55.863870 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.863843 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hf6p\" (UniqueName: \"kubernetes.io/projected/aad276ab-66e3-4cc8-a0a5-6749b625419b-kube-api-access-4hf6p\") pod \"node-resolver-9fdr6\" (UID: \"aad276ab-66e3-4cc8-a0a5-6749b625419b\") " pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:55.864225 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.864206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742z9\" (UniqueName: \"kubernetes.io/projected/2b62be6d-4dc8-4319-8e0a-29e78e268b31-kube-api-access-742z9\") pod \"ovnkube-node-gw9lp\" (UID: \"2b62be6d-4dc8-4319-8e0a-29e78e268b31\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:55.864664 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.864646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bglsw\" (UniqueName: \"kubernetes.io/projected/2323e31a-ed38-40a0-a730-ef6ef0acc26a-kube-api-access-bglsw\") pod \"multus-additional-cni-plugins-mwrvj\" (UID: \"2323e31a-ed38-40a0-a730-ef6ef0acc26a\") " pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:55.935449 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:55.935418 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c7bc3b4c133f1b0578384be47f8d38.slice/crio-3647eeda650a5678f185d743871e568f386ab7249d0e62fcb70dcaee2e837654 WatchSource:0}: Error finding container 3647eeda650a5678f185d743871e568f386ab7249d0e62fcb70dcaee2e837654: Status 404 returned error can't find the container with id 3647eeda650a5678f185d743871e568f386ab7249d0e62fcb70dcaee2e837654 Jul 09 14:13:55.951718 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951693 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-socket-dir-parent\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-k8s-cni-cncf-io\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-netns\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-systemd\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-kube-api-access-55z7n\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-cni-multus\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-netns\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-k8s-cni-cncf-io\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951867 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-systemd\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.951873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysctl-d\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-etc-kubernetes\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-cni-multus\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-socket-dir-parent\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951955 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysctl-d\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-sys\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-etc-kubernetes\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.951992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-var-lib-kubelet\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-sys\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952019 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-var-lib-kubelet\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtzn\" (UniqueName: \"kubernetes.io/projected/e5effc12-cbb1-4745-a99b-9fdab63138af-kube-api-access-jmtzn\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsssq\" (UniqueName: \"kubernetes.io/projected/b950518d-63fa-4cdc-92af-2f6218d8da67-kube-api-access-fsssq\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.952158 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:55.952377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-socket-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-registration-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-kubelet\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-hostroot\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-run\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.952675 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret podName:2cdcb585-98f7-4c44-928e-68b3dc41164e nodeName:}" failed. No retries permitted until 2026-07-09 14:13:56.452632199 +0000 UTC m=+3.141561421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret") pod "global-pull-secret-syncer-bgmqg" (UID: "2cdcb585-98f7-4c44-928e-68b3dc41164e") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-lib-modules\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952751 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-socket-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-cni-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-conf-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-hostroot\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-run\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2cdcb585-98f7-4c44-928e-68b3dc41164e-dbus\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-registration-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.953037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-cni-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-etc-selinux\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952930 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2cdcb585-98f7-4c44-928e-68b3dc41164e-dbus\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952956 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-conf-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.952988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-kubernetes\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-host\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953005 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-kubelet\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-tmp\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-os-release\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-host\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-etc-selinux\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-lib-modules\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-kubernetes\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-os-release\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b950518d-63fa-4cdc-92af-2f6218d8da67-iptables-alerter-script\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.953726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b950518d-63fa-4cdc-92af-2f6218d8da67-host-slash\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b950518d-63fa-4cdc-92af-2f6218d8da67-host-slash\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.953927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-device-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-modprobe-d\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954105 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-device-dir\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysctl-conf\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-tuned\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954106 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b950518d-63fa-4cdc-92af-2f6218d8da67-iptables-alerter-script\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-modprobe-d\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954268 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysconfig\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s22nx\" (UniqueName: \"kubernetes.io/projected/e810efc5-e567-4924-af70-18dd39817937-kube-api-access-s22nx\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954341 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysctl-conf\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.954487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954360 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-sysconfig\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-system-cni-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5effc12-cbb1-4745-a99b-9fdab63138af-cni-binary-copy\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-cni-bin\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-daemon-config\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-multus-certs\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-sys-fs\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-system-cni-dir\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-cnibin\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwmqz\" (UniqueName: \"kubernetes.io/projected/1168fa75-355f-49ac-aabf-50aa7913d802-kube-api-access-cwmqz\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2cdcb585-98f7-4c44-928e-68b3dc41164e-kubelet-config\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.955107 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.954955 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2cdcb585-98f7-4c44-928e-68b3dc41164e-kubelet-config\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:55.955692 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.955504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5effc12-cbb1-4745-a99b-9fdab63138af-cni-binary-copy\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955692 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.955526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5effc12-cbb1-4745-a99b-9fdab63138af-multus-daemon-config\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955692 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.955609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-var-lib-cni-bin\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955692 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.955686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-host-run-multus-certs\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955932 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.955756 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1168fa75-355f-49ac-aabf-50aa7913d802-sys-fs\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:55.955932 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.955772 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:55.955932 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.955819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5effc12-cbb1-4745-a99b-9fdab63138af-cnibin\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.955932 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:55.955852 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs podName:e810efc5-e567-4924-af70-18dd39817937 nodeName:}" failed. No retries permitted until 2026-07-09 14:13:56.455818246 +0000 UTC m=+3.144747467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs") pod "network-metrics-daemon-8jdld" (UID: "e810efc5-e567-4924-af70-18dd39817937") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:55.957347 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.957196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-tmp\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.958777 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.958758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-etc-tuned\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.962864 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.962834 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s22nx\" (UniqueName: \"kubernetes.io/projected/e810efc5-e567-4924-af70-18dd39817937-kube-api-access-s22nx\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:55.962971 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.962868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtzn\" (UniqueName: \"kubernetes.io/projected/e5effc12-cbb1-4745-a99b-9fdab63138af-kube-api-access-jmtzn\") pod \"multus-8954f\" (UID: \"e5effc12-cbb1-4745-a99b-9fdab63138af\") " pod="openshift-multus/multus-8954f" Jul 09 14:13:55.962971 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.962941 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/cb1a8ad6-9600-4f66-b166-1a65f2ef5e45-kube-api-access-55z7n\") pod \"tuned-tc9w9\" (UID: \"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45\") " pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:55.963231 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.963209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsssq\" (UniqueName: \"kubernetes.io/projected/b950518d-63fa-4cdc-92af-2f6218d8da67-kube-api-access-fsssq\") pod \"iptables-alerter-sz4qc\" (UID: \"b950518d-63fa-4cdc-92af-2f6218d8da67\") " pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:55.964515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:55.964498 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwmqz\" (UniqueName: \"kubernetes.io/projected/1168fa75-355f-49ac-aabf-50aa7913d802-kube-api-access-cwmqz\") pod \"aws-ebs-csi-driver-node-t8zhx\" (UID: \"1168fa75-355f-49ac-aabf-50aa7913d802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:56.047285 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.047153 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:13:56.057795 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.057762 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ae436a_86e2_4812_86cd_7ca9b4adc1b5.slice/crio-f0562ec05c4331982bc67e03575de42dfbac3bb96140b15e7f425c9098af4882 WatchSource:0}: Error finding container f0562ec05c4331982bc67e03575de42dfbac3bb96140b15e7f425c9098af4882: Status 404 returned error can't find the container with id f0562ec05c4331982bc67e03575de42dfbac3bb96140b15e7f425c9098af4882 Jul 09 14:13:56.058372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.058352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fdr6" Jul 09 14:13:56.065264 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.065240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" Jul 09 14:13:56.067498 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.067475 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad276ab_66e3_4cc8_a0a5_6749b625419b.slice/crio-be22c34573a8172e0af568a18d388a5632c601fe9d21eaa122dce284084c6889 WatchSource:0}: Error finding container be22c34573a8172e0af568a18d388a5632c601fe9d21eaa122dce284084c6889: Status 404 returned error can't find the container with id be22c34573a8172e0af568a18d388a5632c601fe9d21eaa122dce284084c6889 Jul 09 14:13:56.074422 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.074402 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2323e31a_ed38_40a0_a730_ef6ef0acc26a.slice/crio-e2a4c703cc903b429115722a28438a769d6150ebb68b9d2c3c3629f492802b40 WatchSource:0}: Error finding container e2a4c703cc903b429115722a28438a769d6150ebb68b9d2c3c3629f492802b40: Status 404 returned error can't find the container with id e2a4c703cc903b429115722a28438a769d6150ebb68b9d2c3c3629f492802b40 Jul 09 14:13:56.079555 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.079537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9cbwm" Jul 09 14:13:56.085713 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.085696 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:13:56.088718 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.088690 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0115aff2_b98a_4415_8204_4de41320eb9a.slice/crio-7196de4fd05335a66ab40772e99162f5b8da01d1a571893358aff1928f53694b WatchSource:0}: Error finding container 7196de4fd05335a66ab40772e99162f5b8da01d1a571893358aff1928f53694b: Status 404 returned error can't find the container with id 7196de4fd05335a66ab40772e99162f5b8da01d1a571893358aff1928f53694b Jul 09 14:13:56.095099 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.095076 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b62be6d_4dc8_4319_8e0a_29e78e268b31.slice/crio-c28b633d4c6bfafb9b791c511feaddfe4c0029906f1ea06e9d24febb4577daca WatchSource:0}: Error finding container c28b633d4c6bfafb9b791c511feaddfe4c0029906f1ea06e9d24febb4577daca: Status 404 returned error can't find the container with id c28b633d4c6bfafb9b791c511feaddfe4c0029906f1ea06e9d24febb4577daca Jul 09 14:13:56.103459 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.103437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" Jul 09 14:13:56.111613 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.111592 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1168fa75_355f_49ac_aabf_50aa7913d802.slice/crio-0698782250d81dcc418bce06df5d0e931bd9d0a89377be94d862d48fe426af35 WatchSource:0}: Error finding container 0698782250d81dcc418bce06df5d0e931bd9d0a89377be94d862d48fe426af35: Status 404 returned error can't find the container with id 0698782250d81dcc418bce06df5d0e931bd9d0a89377be94d862d48fe426af35 Jul 09 14:13:56.124189 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.124162 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" Jul 09 14:13:56.130525 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.130497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8954f" Jul 09 14:13:56.132544 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.132520 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1a8ad6_9600_4f66_b166_1a65f2ef5e45.slice/crio-9ace78a8e5c164eda187e5b122bd7056247242235876f46947ec344deeee669f WatchSource:0}: Error finding container 9ace78a8e5c164eda187e5b122bd7056247242235876f46947ec344deeee669f: Status 404 returned error can't find the container with id 9ace78a8e5c164eda187e5b122bd7056247242235876f46947ec344deeee669f Jul 09 14:13:56.136685 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.136667 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sz4qc" Jul 09 14:13:56.140292 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.140271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5effc12_cbb1_4745_a99b_9fdab63138af.slice/crio-322c4b4bf1f42b37f3cb33243455b32a916f69918bfe1faedbf425659ab1fa05 WatchSource:0}: Error finding container 322c4b4bf1f42b37f3cb33243455b32a916f69918bfe1faedbf425659ab1fa05: Status 404 returned error can't find the container with id 322c4b4bf1f42b37f3cb33243455b32a916f69918bfe1faedbf425659ab1fa05 Jul 09 14:13:56.144898 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.144806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8954f" event={"ID":"e5effc12-cbb1-4745-a99b-9fdab63138af","Type":"ContainerStarted","Data":"322c4b4bf1f42b37f3cb33243455b32a916f69918bfe1faedbf425659ab1fa05"} Jul 09 14:13:56.145987 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.145953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" event={"ID":"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45","Type":"ContainerStarted","Data":"9ace78a8e5c164eda187e5b122bd7056247242235876f46947ec344deeee669f"} Jul 09 14:13:56.147064 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.147034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" event={"ID":"1168fa75-355f-49ac-aabf-50aa7913d802","Type":"ContainerStarted","Data":"0698782250d81dcc418bce06df5d0e931bd9d0a89377be94d862d48fe426af35"} Jul 09 14:13:56.147310 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:13:56.147270 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb950518d_63fa_4cdc_92af_2f6218d8da67.slice/crio-aecd7f84b1c96b7bbc878a3178a6248ee1b5a56e9b9b96862801c652d939bb13 WatchSource:0}: Error finding container aecd7f84b1c96b7bbc878a3178a6248ee1b5a56e9b9b96862801c652d939bb13: Status 404 returned error can't find the container with id aecd7f84b1c96b7bbc878a3178a6248ee1b5a56e9b9b96862801c652d939bb13 Jul 09 14:13:56.148128 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.148108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerStarted","Data":"e2a4c703cc903b429115722a28438a769d6150ebb68b9d2c3c3629f492802b40"} Jul 09 14:13:56.149300 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.149267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fdr6" event={"ID":"aad276ab-66e3-4cc8-a0a5-6749b625419b","Type":"ContainerStarted","Data":"be22c34573a8172e0af568a18d388a5632c601fe9d21eaa122dce284084c6889"} Jul 09 14:13:56.150491 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.150471 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"34b43d5f526b7974ac738b5aaab0d959","Type":"ContainerStarted","Data":"11f6197a9c6d60dc13233f62a18034112c44099b1bbd0a43ab6f56c79bedf439"} Jul 09 14:13:56.151670 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.151650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"c28b633d4c6bfafb9b791c511feaddfe4c0029906f1ea06e9d24febb4577daca"} Jul 09 14:13:56.152847 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.152826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9cbwm" event={"ID":"0115aff2-b98a-4415-8204-4de41320eb9a","Type":"ContainerStarted","Data":"7196de4fd05335a66ab40772e99162f5b8da01d1a571893358aff1928f53694b"} Jul 09 14:13:56.154009 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.153990 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d22n5" event={"ID":"86ae436a-86e2-4812-86cd-7ca9b4adc1b5","Type":"ContainerStarted","Data":"f0562ec05c4331982bc67e03575de42dfbac3bb96140b15e7f425c9098af4882"} Jul 09 14:13:56.155039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.155019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"a1c7bc3b4c133f1b0578384be47f8d38","Type":"ContainerStarted","Data":"3647eeda650a5678f185d743871e568f386ab7249d0e62fcb70dcaee2e837654"} Jul 09 14:13:56.458553 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.458468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:56.458553 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.458530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.458560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458672 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458706 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458730 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458742 2575 projected.go:196] Error preparing data for projected volume kube-api-access-r9tmd for pod openshift-network-diagnostics/network-check-target-lfhcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458739 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458758 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs podName:e810efc5-e567-4924-af70-18dd39817937 nodeName:}" failed. No retries permitted until 2026-07-09 14:13:57.458736286 +0000 UTC m=+4.147665522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs") pod "network-metrics-daemon-8jdld" (UID: "e810efc5-e567-4924-af70-18dd39817937") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458796 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd podName:50732da1-a962-4c24-8c2e-f71c931e04c1 nodeName:}" failed. No retries permitted until 2026-07-09 14:13:57.458780252 +0000 UTC m=+4.147709505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9tmd" (UniqueName: "kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd") pod "network-check-target-lfhcd" (UID: "50732da1-a962-4c24-8c2e-f71c931e04c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:56.458824 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.458832 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret podName:2cdcb585-98f7-4c44-928e-68b3dc41164e nodeName:}" failed. No retries permitted until 2026-07-09 14:13:57.458806552 +0000 UTC m=+4.147735774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret") pod "global-pull-secret-syncer-bgmqg" (UID: "2cdcb585-98f7-4c44-928e-68b3dc41164e") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:56.786497 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.786373 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:08:56 +0000 UTC" deadline="2028-02-13 07:12:14.22602439 +0000 UTC" Jul 09 14:13:56.786497 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.786440 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14008h58m17.439589554s" Jul 09 14:13:56.880858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:56.880199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:56.880858 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:56.880362 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:13:57.170702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.170607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sz4qc" event={"ID":"b950518d-63fa-4cdc-92af-2f6218d8da67","Type":"ContainerStarted","Data":"aecd7f84b1c96b7bbc878a3178a6248ee1b5a56e9b9b96862801c652d939bb13"} Jul 09 14:13:57.469430 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.469341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:57.469430 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.469409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:57.469669 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.469441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:57.469669 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.469620 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:13:57.469669 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.469640 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:13:57.469669 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.469652 2575 projected.go:196] Error preparing data for projected volume kube-api-access-r9tmd for pod openshift-network-diagnostics/network-check-target-lfhcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:57.469888 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.469717 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd podName:50732da1-a962-4c24-8c2e-f71c931e04c1 nodeName:}" failed. No retries permitted until 2026-07-09 14:13:59.469696913 +0000 UTC m=+6.158626158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9tmd" (UniqueName: "kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd") pod "network-check-target-lfhcd" (UID: "50732da1-a962-4c24-8c2e-f71c931e04c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:57.470647 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.470221 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:57.470647 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.470277 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs podName:e810efc5-e567-4924-af70-18dd39817937 nodeName:}" failed. No retries permitted until 2026-07-09 14:13:59.470261401 +0000 UTC m=+6.159190624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs") pod "network-metrics-daemon-8jdld" (UID: "e810efc5-e567-4924-af70-18dd39817937") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:57.470647 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.470361 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:57.470647 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.470408 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret podName:2cdcb585-98f7-4c44-928e-68b3dc41164e nodeName:}" failed. No retries permitted until 2026-07-09 14:13:59.470393742 +0000 UTC m=+6.159322966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret") pod "global-pull-secret-syncer-bgmqg" (UID: "2cdcb585-98f7-4c44-928e-68b3dc41164e") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:57.787514 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.787406 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:08:56 +0000 UTC" deadline="2027-12-13 08:58:14.641116523 +0000 UTC" Jul 09 14:13:57.787514 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.787454 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12522h44m16.853668504s" Jul 09 14:13:57.883724 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.882665 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:57.883724 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.882960 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:13:57.883724 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:57.883510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:57.883724 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:57.883614 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:13:58.880337 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:58.880291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:58.880978 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:58.880442 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:13:59.490767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:59.489898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:59.490767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:59.489963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:13:59.490767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:59.490030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:59.490767 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490166 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:59.490767 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490237 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs podName:e810efc5-e567-4924-af70-18dd39817937 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:03.490216619 +0000 UTC m=+10.179145842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs") pod "network-metrics-daemon-8jdld" (UID: "e810efc5-e567-4924-af70-18dd39817937") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:13:59.490767 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490680 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:59.490767 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490741 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret podName:2cdcb585-98f7-4c44-928e-68b3dc41164e nodeName:}" failed. No retries permitted until 2026-07-09 14:14:03.49072517 +0000 UTC m=+10.179654391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret") pod "global-pull-secret-syncer-bgmqg" (UID: "2cdcb585-98f7-4c44-928e-68b3dc41164e") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:13:59.491379 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490800 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:13:59.491379 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490822 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:13:59.491379 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490835 2575 projected.go:196] Error preparing data for projected volume kube-api-access-r9tmd for pod openshift-network-diagnostics/network-check-target-lfhcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:59.491379 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.490889 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd podName:50732da1-a962-4c24-8c2e-f71c931e04c1 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:03.4908766 +0000 UTC m=+10.179805840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9tmd" (UniqueName: "kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd") pod "network-check-target-lfhcd" (UID: "50732da1-a962-4c24-8c2e-f71c931e04c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:13:59.880286 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:59.880189 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:13:59.880524 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.880369 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:13:59.880524 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:13:59.880205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:13:59.880524 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:13:59.880481 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:00.879870 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:00.879833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:00.880068 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:00.879978 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:01.880617 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:01.880280 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:01.880617 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:01.880457 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:01.881146 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:01.880283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:01.881146 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:01.880930 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:02.880696 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:02.880577 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:02.881181 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:02.880725 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:03.525848 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:03.525554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:03.525848 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:03.525615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:03.525848 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:03.525683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:03.525848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525748 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:03.525848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525795 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:03.525848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525829 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret podName:2cdcb585-98f7-4c44-928e-68b3dc41164e nodeName:}" failed. No retries permitted until 2026-07-09 14:14:11.525804634 +0000 UTC m=+18.214733858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret") pod "global-pull-secret-syncer-bgmqg" (UID: "2cdcb585-98f7-4c44-928e-68b3dc41164e") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:03.525848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525852 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs podName:e810efc5-e567-4924-af70-18dd39817937 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:11.525840122 +0000 UTC m=+18.214769360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs") pod "network-metrics-daemon-8jdld" (UID: "e810efc5-e567-4924-af70-18dd39817937") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:03.526345 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525793 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:03.526345 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525902 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:03.526345 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525919 2575 projected.go:196] Error preparing data for projected volume kube-api-access-r9tmd for pod openshift-network-diagnostics/network-check-target-lfhcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:03.526345 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.525985 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd podName:50732da1-a962-4c24-8c2e-f71c931e04c1 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:11.525967686 +0000 UTC m=+18.214896916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9tmd" (UniqueName: "kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd") pod "network-check-target-lfhcd" (UID: "50732da1-a962-4c24-8c2e-f71c931e04c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:03.881086 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:03.880746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:03.881086 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.880882 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:03.881086 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:03.881015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:03.881712 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:03.881156 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:04.880628 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:04.880589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:04.880825 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:04.880738 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:05.880671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:05.880573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:05.881181 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:05.880736 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:05.881181 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:05.880759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:05.881181 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:05.880877 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:06.880427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:06.880392 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:06.880628 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:06.880530 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:07.880547 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:07.880428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:07.881012 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:07.880581 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:07.881012 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:07.880665 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:07.881012 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:07.880778 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:08.880237 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:08.880103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:08.880354 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:08.880315 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:09.206807 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:09.206720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"a1c7bc3b4c133f1b0578384be47f8d38","Type":"ContainerStarted","Data":"27e7e92ef1ce25044e038bc111958bf1c9a53b2cda02c4641b852206f748aa97"} Jul 09 14:14:09.208561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:09.208535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" event={"ID":"cb1a8ad6-9600-4f66-b166-1a65f2ef5e45","Type":"ContainerStarted","Data":"b07a11d8173e094d36d12f541495a245ae81169deb570eccc06dbb4c6e7128dc"} Jul 09 14:14:09.222020 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:09.221968 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" podStartSLOduration=13.221950358 podStartE2EDuration="13.221950358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:09.221598075 +0000 UTC m=+15.910527334" watchObservedRunningTime="2026-07-09 14:14:09.221950358 +0000 UTC m=+15.910879599" Jul 09 14:14:09.237406 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:09.237351 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tc9w9" podStartSLOduration=1.659803143 podStartE2EDuration="14.237330416s" totalImagesPullingTime="12.577527273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:09.237259059 +0000 UTC m=+15.926188326" watchObservedRunningTime="2026-07-09 14:14:09.237330416 +0000 UTC m=+15.926259656" Jul 09 14:14:09.881502 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:09.881432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:09.881606 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:09.881577 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:09.881891 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:09.881703 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:09.881891 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:09.881830 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:10.212921 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.212844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" event={"ID":"1168fa75-355f-49ac-aabf-50aa7913d802","Type":"ContainerStarted","Data":"63e35b738b7a3a35d4be4a804949928aac4321e9b30866b2dfd7a53b3e2b8d8c"} Jul 09 14:14:10.214654 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.214626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerStarted","Data":"08706fc84916ac189450ac0bde752530fa9f2163a8333a32bd91a2a292587790"} Jul 09 14:14:10.216352 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.216323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fdr6" event={"ID":"aad276ab-66e3-4cc8-a0a5-6749b625419b","Type":"ContainerStarted","Data":"b55d2e08cc11dc3d41b44d51cd5f56a2d0dbcfa86dc3b6856de53d476efa189a"} Jul 09 14:14:10.218036 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.218013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"34b43d5f526b7974ac738b5aaab0d959","Type":"ContainerStarted","Data":"eeb284366705cbb0129d54ce62dd856acdfa86ff9b651407b21303b0fe766e5d"} Jul 09 14:14:10.219930 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.219895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9cbwm" event={"ID":"0115aff2-b98a-4415-8204-4de41320eb9a","Type":"ContainerStarted","Data":"7c5b8da278ae073b27027a948637ebaa7892b8a2f2972a2735d652071844929c"} Jul 09 14:14:10.221543 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.221523 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d22n5" event={"ID":"86ae436a-86e2-4812-86cd-7ca9b4adc1b5","Type":"ContainerStarted","Data":"4ba71aab3e60185797ca5c1a03478c6b2818d3c0c4c026f6dbf0fdbc16178b7a"} Jul 09 14:14:10.255095 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.255038 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9fdr6" podStartSLOduration=3.629748284 podStartE2EDuration="16.255019915s" totalImagesPullingTime="12.625271631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:10.254608296 +0000 UTC m=+16.943537555" watchObservedRunningTime="2026-07-09 14:14:10.255019915 +0000 UTC m=+16.943949153" Jul 09 14:14:10.286242 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.286052 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d22n5" podStartSLOduration=3.723215846 podStartE2EDuration="16.286036796s" totalImagesPullingTime="12.56282095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:10.285760292 +0000 UTC m=+16.974689531" watchObservedRunningTime="2026-07-09 14:14:10.286036796 +0000 UTC m=+16.974966088" Jul 09 14:14:10.300102 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.300045 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9cbwm" podStartSLOduration=2.694093434 podStartE2EDuration="15.300025029s" totalImagesPullingTime="12.605931595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:10.299825981 +0000 UTC m=+16.988755255" watchObservedRunningTime="2026-07-09 14:14:10.300025029 +0000 UTC m=+16.988954273" Jul 09 14:14:10.880290 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:10.880253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:10.880596 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:10.880388 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:11.226259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.225947 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sz4qc" event={"ID":"b950518d-63fa-4cdc-92af-2f6218d8da67","Type":"ContainerStarted","Data":"0eaee18dd19164303f9fae27e600fc2a07cf4f1ade37710e661064fec7056cb3"} Jul 09 14:14:11.241832 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.241774 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sz4qc" podStartSLOduration=3.692086332 podStartE2EDuration="16.241753278s" totalImagesPullingTime="12.549666946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:11.241057681 +0000 UTC m=+17.929986921" watchObservedRunningTime="2026-07-09 14:14:11.241753278 +0000 UTC m=+17.930682518" Jul 09 14:14:11.593557 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.593456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:11.593557 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.593528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:11.593744 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.593561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:11.593744 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593606 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:11.593744 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593670 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:11.593744 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593678 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:11.593744 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593696 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:11.593744 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593705 2575 projected.go:196] Error preparing data for projected volume kube-api-access-r9tmd for pod openshift-network-diagnostics/network-check-target-lfhcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:11.593744 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593674 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs podName:e810efc5-e567-4924-af70-18dd39817937 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:27.593657285 +0000 UTC m=+34.282586501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs") pod "network-metrics-daemon-8jdld" (UID: "e810efc5-e567-4924-af70-18dd39817937") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:11.593975 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593750 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret podName:2cdcb585-98f7-4c44-928e-68b3dc41164e nodeName:}" failed. No retries permitted until 2026-07-09 14:14:27.593735143 +0000 UTC m=+34.282664364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret") pod "global-pull-secret-syncer-bgmqg" (UID: "2cdcb585-98f7-4c44-928e-68b3dc41164e") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:11.593975 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.593764 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd podName:50732da1-a962-4c24-8c2e-f71c931e04c1 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:27.593756906 +0000 UTC m=+34.282686123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9tmd" (UniqueName: "kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd") pod "network-check-target-lfhcd" (UID: "50732da1-a962-4c24-8c2e-f71c931e04c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:11.617100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.617072 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:14:11.617663 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.617642 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:14:11.880741 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.880656 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:11.880741 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:11.880685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:11.880945 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.880781 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:11.881000 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:11.880973 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:12.879803 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:12.879767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:12.880262 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:12.879886 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:13.231515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:13.231477 2575 generic.go:356] "Generic (PLEG): container finished" podID="2323e31a-ed38-40a0-a730-ef6ef0acc26a" containerID="08706fc84916ac189450ac0bde752530fa9f2163a8333a32bd91a2a292587790" exitCode=0 Jul 09 14:14:13.231515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:13.231508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerDied","Data":"08706fc84916ac189450ac0bde752530fa9f2163a8333a32bd91a2a292587790"} Jul 09 14:14:13.233195 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:13.233171 2575 generic.go:356] "Generic (PLEG): container finished" podID="34b43d5f526b7974ac738b5aaab0d959" containerID="eeb284366705cbb0129d54ce62dd856acdfa86ff9b651407b21303b0fe766e5d" exitCode=0 Jul 09 14:14:13.233322 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:13.233201 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"34b43d5f526b7974ac738b5aaab0d959","Type":"ContainerDied","Data":"eeb284366705cbb0129d54ce62dd856acdfa86ff9b651407b21303b0fe766e5d"} Jul 09 14:14:13.233322 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:13.233259 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 14:14:13.881016 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:13.880979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:13.881634 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:13.881105 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:13.881634 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:13.881166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:13.881634 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:13.881298 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:14.879728 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:14.879692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:14.879886 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:14.879802 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:15.879733 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:15.879702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:15.880327 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:15.879702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:15.880327 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:15.879830 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:15.880327 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:15.879890 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:16.880293 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:16.880258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:16.880771 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:16.880381 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:17.737622 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:17.736899 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 14:14:17.806101 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:17.805987 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T14:14:17.737058012Z","UUID":"f3d3a76b-3a2c-4e61-afdb-14296393da8d","Handler":null,"Name":"","Endpoint":""} Jul 09 14:14:17.808659 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:17.808563 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 14:14:17.808659 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:17.808605 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 14:14:17.880624 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:17.880549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:17.880624 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:17.880579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:17.881193 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:17.880690 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:17.881193 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:17.880913 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:18.246815 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.246402 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8954f" event={"ID":"e5effc12-cbb1-4745-a99b-9fdab63138af","Type":"ContainerStarted","Data":"7c3885d5d74d860bcaef99aa604253272b7444784285c9e6d0ae694ecd6bdc42"} Jul 09 14:14:18.248458 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.248424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" event={"ID":"1168fa75-355f-49ac-aabf-50aa7913d802","Type":"ContainerStarted","Data":"297b6bc2f7b4e7f225e123203a221c2c3532c6fbabb0b355f4deeb0ea601f72f"} Jul 09 14:14:18.250995 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.250942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"34b43d5f526b7974ac738b5aaab0d959","Type":"ContainerStarted","Data":"4cf59c3c7f71486fe62d70272ef06ed738a8b313de0a0711c91652f118169abf"} Jul 09 14:14:18.254668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.254634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"c70be409ac4564bc8d37198e997204bbfa0169f064f9a92fd68fa2c392009695"} Jul 09 14:14:18.254767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.254708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"05098b1d7d0f2b4868dc74c2129bbf80c8dfe91df25a1f1241c78976e9248d0a"} Jul 09 14:14:18.254767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.254758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"4fb49c43fe5d3901a26498b1b02dd2f248f726de4c714899c68b554667f26768"} Jul 09 14:14:18.254879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.254806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"f13bdf706498714620a5c965bea05c6c335c470cf6c11a4cfdb81fc70935f435"} Jul 09 14:14:18.254879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.254862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"0792dcbedb3ba1d37a326d8023744e0841d2324bfdff61214f1e26cacadb9ee1"} Jul 09 14:14:18.254989 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.254896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"cbe7b2a3928f6c7425f1a9f478b29bd39f73d27982a7bc89cb3d22284abad057"} Jul 09 14:14:18.267291 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.267253 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8954f" podStartSLOduration=1.8420144729999999 podStartE2EDuration="23.267239307s" totalImagesPullingTime="21.425224834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:18.266819622 +0000 UTC m=+24.955748862" watchObservedRunningTime="2026-07-09 14:14:18.267239307 +0000 UTC m=+24.956168540" Jul 09 14:14:18.283718 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.283668 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" podStartSLOduration=22.283651598 podStartE2EDuration="22.283651598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:18.282835864 +0000 UTC m=+24.971765129" watchObservedRunningTime="2026-07-09 14:14:18.283651598 +0000 UTC m=+24.972580839" Jul 09 14:14:18.879895 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:18.879744 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:18.880117 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:18.879965 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:19.259483 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:19.259448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" event={"ID":"1168fa75-355f-49ac-aabf-50aa7913d802","Type":"ContainerStarted","Data":"8cb89082f98b45102f6805b21373e07657f7f9c1865aee32f24ae1983b004ea3"} Jul 09 14:14:19.880532 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:19.880496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:19.880699 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:19.880496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:19.880699 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:19.880604 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:19.880775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:19.880721 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:20.880639 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:20.880599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:20.881029 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:20.880702 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:21.271759 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:21.271730 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"3c42aa48f552e0bb01114205ad92a9bdb8979d175e94ab34305af0023c1b49bb"} Jul 09 14:14:21.880001 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:21.879962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:21.880166 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:21.880017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:21.880166 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:21.880114 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:21.880301 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:21.880275 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:22.879883 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:22.879806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:22.880501 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:22.879909 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:23.278149 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.277985 2575 generic.go:356] "Generic (PLEG): container finished" podID="2323e31a-ed38-40a0-a730-ef6ef0acc26a" containerID="769c38d4d5cc029c63326dd3c7891788735cb0e873e47086a1ef151a44aa0978" exitCode=0 Jul 09 14:14:23.278334 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.278057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerDied","Data":"769c38d4d5cc029c63326dd3c7891788735cb0e873e47086a1ef151a44aa0978"} Jul 09 14:14:23.281854 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.281831 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" event={"ID":"2b62be6d-4dc8-4319-8e0a-29e78e268b31","Type":"ContainerStarted","Data":"32cab72384c77d94a4526174d68d5d097f4ac63a5b445bcab2db82e32006b0fc"} Jul 09 14:14:23.282112 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.282089 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:14:23.282205 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.282188 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:14:23.297085 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.297062 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:14:23.297198 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.297156 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:14:23.306708 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.306669 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t8zhx" podStartSLOduration=6.723845837 podStartE2EDuration="28.30665705s" totalImagesPullingTime="21.582811213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:19.285662517 +0000 UTC m=+25.974591766" watchObservedRunningTime="2026-07-09 14:14:23.30665705 +0000 UTC m=+29.995586290" Jul 09 14:14:23.338780 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.338723 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" podStartSLOduration=6.86546321 podStartE2EDuration="28.338710573s" totalImagesPullingTime="21.473247363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:23.337836148 +0000 UTC m=+30.026765386" watchObservedRunningTime="2026-07-09 14:14:23.338710573 +0000 UTC m=+30.027639812" Jul 09 14:14:23.747885 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.747847 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:14:23.748163 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.747962 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 14:14:23.748415 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.748386 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d22n5" Jul 09 14:14:23.880551 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.880515 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:23.881050 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:23.880625 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:23.881050 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:23.880681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:23.881050 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:23.880768 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:24.186763 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:24.186729 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:14:24.880471 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:24.880440 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:24.880630 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:24.880543 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:24.936179 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:24.936095 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bgmqg"] Jul 09 14:14:24.936328 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:24.936215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:24.936328 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:24.936318 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:24.939272 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:24.939249 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jdld"] Jul 09 14:14:24.939403 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:24.939356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:24.939490 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:24.939466 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:24.939870 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:24.939852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfhcd"] Jul 09 14:14:25.287892 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:25.287861 2575 generic.go:356] "Generic (PLEG): container finished" podID="2323e31a-ed38-40a0-a730-ef6ef0acc26a" containerID="9d728aff6a143b7b43a0e4360e0df79657142b74363b17de74ca3975e3a5d9ba" exitCode=0 Jul 09 14:14:25.288049 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:25.287942 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:25.288049 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:25.287943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerDied","Data":"9d728aff6a143b7b43a0e4360e0df79657142b74363b17de74ca3975e3a5d9ba"} Jul 09 14:14:25.288049 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:25.288041 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:26.879674 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:26.879636 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:26.879674 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:26.879673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:26.880114 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:26.879997 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:26.882247 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:26.880200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:26.882247 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:26.880431 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:26.882247 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:26.880573 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:27.294652 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:27.294617 2575 generic.go:356] "Generic (PLEG): container finished" podID="2323e31a-ed38-40a0-a730-ef6ef0acc26a" containerID="d4f810320de798a427b67ae9b6eef52a2e7b5764a04f1b7bbc5bd2fe0a563ee3" exitCode=0 Jul 09 14:14:27.294799 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:27.294669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerDied","Data":"d4f810320de798a427b67ae9b6eef52a2e7b5764a04f1b7bbc5bd2fe0a563ee3"} Jul 09 14:14:27.617417 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:27.617321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:27.617417 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:27.617365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:27.617416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617457 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617513 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret podName:2cdcb585-98f7-4c44-928e-68b3dc41164e nodeName:}" failed. No retries permitted until 2026-07-09 14:14:59.617499778 +0000 UTC m=+66.306428995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret") pod "global-pull-secret-syncer-bgmqg" (UID: "2cdcb585-98f7-4c44-928e-68b3dc41164e") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617519 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617530 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617549 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617559 2575 projected.go:196] Error preparing data for projected volume kube-api-access-r9tmd for pod openshift-network-diagnostics/network-check-target-lfhcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617565 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs podName:e810efc5-e567-4924-af70-18dd39817937 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:59.617549773 +0000 UTC m=+66.306478991 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs") pod "network-metrics-daemon-8jdld" (UID: "e810efc5-e567-4924-af70-18dd39817937") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:27.617635 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:27.617596 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd podName:50732da1-a962-4c24-8c2e-f71c931e04c1 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:59.617586136 +0000 UTC m=+66.306515353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9tmd" (UniqueName: "kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd") pod "network-check-target-lfhcd" (UID: "50732da1-a962-4c24-8c2e-f71c931e04c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:28.880488 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:28.880307 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:28.880947 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:28.880307 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:28.880947 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:28.880324 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:28.880947 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:28.880582 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jdld" podUID="e810efc5-e567-4924-af70-18dd39817937" Jul 09 14:14:28.880947 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:28.880638 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfhcd" podUID="50732da1-a962-4c24-8c2e-f71c931e04c1" Jul 09 14:14:28.880947 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:28.880729 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgmqg" podUID="2cdcb585-98f7-4c44-928e-68b3dc41164e" Jul 09 14:14:30.147927 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.147899 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeReady" Jul 09 14:14:30.148320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.148055 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 14:14:30.189363 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.189310 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c47c5d5b5-pnpfj"] Jul 09 14:14:30.192922 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.192902 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk"] Jul 09 14:14:30.193071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.193048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.195820 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.195798 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-57dt4\"" Jul 09 14:14:30.195915 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.195800 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 14:14:30.195915 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.195798 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 14:14:30.195915 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.195885 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 14:14:30.197721 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.197701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" Jul 09 14:14:30.200614 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.200569 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.200614 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.200588 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nlrhj\"" Jul 09 14:14:30.200614 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.200573 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.202605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.202586 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 14:14:30.204193 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.204171 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk"] Jul 09 14:14:30.211946 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.211929 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c47c5d5b5-pnpfj"] Jul 09 14:14:30.213529 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.213509 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92"] Jul 09 14:14:30.217520 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.217504 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7"] Jul 09 14:14:30.217657 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.217642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.220926 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.220908 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-wxw4j"] Jul 09 14:14:30.221025 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.221002 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.222428 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.222408 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5m487\"" Jul 09 14:14:30.222791 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.222771 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.222791 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.222784 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 14:14:30.222791 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.222793 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.223080 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.223062 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 14:14:30.224431 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.224411 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qptsb"] Jul 09 14:14:30.224522 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.224492 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 14:14:30.224603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.224587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.224789 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.224757 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 14:14:30.224870 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.224809 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-twt95\"" Jul 09 14:14:30.225115 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.225047 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.225115 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.225102 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.228176 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.228158 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74555698d-dkf2p"] Jul 09 14:14:30.228323 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.228301 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.229561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.229536 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.229561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.229553 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.229716 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.229623 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 14:14:30.229865 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.229839 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-v58fc\"" Jul 09 14:14:30.229943 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.229906 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 14:14:30.231672 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.231650 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b8fdl"] Jul 09 14:14:30.231784 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.231759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.234096 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.233836 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5ht22\"" Jul 09 14:14:30.236616 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.236583 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.236697 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.236642 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.236903 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.236886 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 14:14:30.237035 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.237016 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 14:14:30.237177 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.237150 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 14:14:30.238118 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.237847 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.238619 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.238602 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 14:14:30.238803 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.238785 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.238931 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.238620 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b"] Jul 09 14:14:30.239284 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.238682 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.240047 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.240025 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 14:14:30.240397 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.240089 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 14:14:30.242688 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.242668 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w749s\"" Jul 09 14:14:30.243113 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.243084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 14:14:30.246948 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.246785 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kggvr\"" Jul 09 14:14:30.247075 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.247018 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 14:14:30.247630 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.247595 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm"] Jul 09 14:14:30.248504 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.248485 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.249451 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.248976 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 14:14:30.249566 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.249544 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 14:14:30.253185 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.253170 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r"] Jul 09 14:14:30.253439 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.253422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" Jul 09 14:14:30.257042 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.257023 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.257137 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.257074 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.257772 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.257754 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 14:14:30.257881 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.257852 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.257942 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.257901 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 14:14:30.258130 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.258112 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vbgn4\"" Jul 09 14:14:30.258915 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.258898 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rxfgm\"" Jul 09 14:14:30.259028 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.258910 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.260757 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.260739 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-866jh"] Jul 09 14:14:30.261770 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.260885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:30.264718 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.264700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 14:14:30.264819 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.264748 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sglr8"] Jul 09 14:14:30.264887 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.264862 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.264943 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.264906 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.265157 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.265141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:30.265157 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.265153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b5plj\"" Jul 09 14:14:30.267498 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.267479 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 14:14:30.267797 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.267664 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v24rb\"" Jul 09 14:14:30.268034 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.268017 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 14:14:30.268211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.268189 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-547b9bc97d-r5mw4"] Jul 09 14:14:30.268324 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.268309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.271185 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.271135 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 14:14:30.271412 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.271393 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 14:14:30.271651 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.271634 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4fkhx\"" Jul 09 14:14:30.272445 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.272428 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.274789 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.274830 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.274872 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-wxw4j"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.274895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-866jh"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.274922 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547b9bc97d-r5mw4"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.274943 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8fdl"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.274985 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74555698d-dkf2p"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.275005 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7"] Jul 09 14:14:30.275031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.275028 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm"] Jul 09 14:14:30.275462 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.275051 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sglr8"] Jul 09 14:14:30.275462 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.275077 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r"] Jul 09 14:14:30.275462 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.275196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.275462 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.275367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qptsb"] Jul 09 14:14:30.293535 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.293511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:30.337186 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d25ba2-c533-44bd-be10-c9da17f3929c-tmp-dir\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.337320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mzn\" (UniqueName: \"kubernetes.io/projected/5a3562bb-92a7-4cc1-91bf-9df604d84f92-kube-api-access-w6mzn\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.337320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskw4\" (UniqueName: \"kubernetes.io/projected/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-kube-api-access-cskw4\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.337320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3506c053-ea2a-4484-b5c3-369c03d8baed-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.337320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-stats-auth\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.337444 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3506c053-ea2a-4484-b5c3-369c03d8baed-config\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.337444 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66bca9b3-d1e5-475b-80a2-824250d8ae0f-serving-cert\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.337444 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrscx\" (UniqueName: \"kubernetes.io/projected/7e281907-2c2d-4944-8736-22f20ce41bb7-kube-api-access-wrscx\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.337444 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66bca9b3-d1e5-475b-80a2-824250d8ae0f-config\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.337562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh4nj\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-kube-api-access-gh4nj\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.337562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-registry-certificates\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-installation-pull-secrets\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-bound-sa-token\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sk42\" (UniqueName: \"kubernetes.io/projected/3506c053-ea2a-4484-b5c3-369c03d8baed-kube-api-access-5sk42\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.337562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3562bb-92a7-4cc1-91bf-9df604d84f92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a3562bb-92a7-4cc1-91bf-9df604d84f92-serving-cert\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5a3562bb-92a7-4cc1-91bf-9df604d84f92-snapshots\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnx7j\" (UniqueName: \"kubernetes.io/projected/7f992406-0c0a-420d-872d-b3a3dd9448a5-kube-api-access-dnx7j\") pod \"volume-data-source-validator-78495597f6-pkfdk\" (UID: \"7f992406-0c0a-420d-872d-b3a3dd9448a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3562bb-92a7-4cc1-91bf-9df604d84f92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66bca9b3-d1e5-475b-80a2-824250d8ae0f-trusted-ca\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.337755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-image-registry-private-configuration\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3562bb-92a7-4cc1-91bf-9df604d84f92-tmp\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94d25ba2-c533-44bd-be10-c9da17f3929c-config-volume\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d1a01d-bbcc-490c-a195-09158937207f-ca-trust-extracted\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfvk\" (UniqueName: \"kubernetes.io/projected/66bca9b3-d1e5-475b-80a2-824250d8ae0f-kube-api-access-khfvk\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4sk\" (UniqueName: \"kubernetes.io/projected/a9284783-b78b-4612-96b7-38fb17d6e4b4-kube-api-access-zw4sk\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9284783-b78b-4612-96b7-38fb17d6e4b4-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-trusted-ca\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.337994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzlq\" (UniqueName: \"kubernetes.io/projected/94d25ba2-c533-44bd-be10-c9da17f3929c-kube-api-access-srzlq\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.338313 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.337997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-default-certificate\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.393091 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.393049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:30.439035 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66bca9b3-d1e5-475b-80a2-824250d8ae0f-config\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.439035 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv62d\" (UniqueName: \"kubernetes.io/projected/1055a6aa-997c-447b-ab11-452a2639dbd8-kube-api-access-sv62d\") pod \"ingress-canary-sglr8\" (UID: \"1055a6aa-997c-447b-ab11-452a2639dbd8\") " pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh4nj\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-kube-api-access-gh4nj\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-trusted-ca\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-registry-certificates\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-installation-pull-secrets\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-bound-sa-token\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sk42\" (UniqueName: \"kubernetes.io/projected/3506c053-ea2a-4484-b5c3-369c03d8baed-kube-api-access-5sk42\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.439289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1055a6aa-997c-447b-ab11-452a2639dbd8-cert\") pod \"ingress-canary-sglr8\" (UID: \"1055a6aa-997c-447b-ab11-452a2639dbd8\") " pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqrd\" (UniqueName: \"kubernetes.io/projected/07d2bb68-32a6-4a7e-a1a7-21ad23767f8c-kube-api-access-lbqrd\") pod \"network-check-source-869dcfcdcd-27tzm\" (UID: \"07d2bb68-32a6-4a7e-a1a7-21ad23767f8c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3562bb-92a7-4cc1-91bf-9df604d84f92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a3562bb-92a7-4cc1-91bf-9df604d84f92-serving-cert\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5a3562bb-92a7-4cc1-91bf-9df604d84f92-snapshots\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnx7j\" (UniqueName: \"kubernetes.io/projected/7f992406-0c0a-420d-872d-b3a3dd9448a5-kube-api-access-dnx7j\") pod \"volume-data-source-validator-78495597f6-pkfdk\" (UID: \"7f992406-0c0a-420d-872d-b3a3dd9448a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3562bb-92a7-4cc1-91bf-9df604d84f92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66bca9b3-d1e5-475b-80a2-824250d8ae0f-trusted-ca\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-image-registry-private-configuration\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.439719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3562bb-92a7-4cc1-91bf-9df604d84f92-tmp\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94d25ba2-c533-44bd-be10-c9da17f3929c-config-volume\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d1a01d-bbcc-490c-a195-09158937207f-ca-trust-extracted\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfvk\" (UniqueName: \"kubernetes.io/projected/66bca9b3-d1e5-475b-80a2-824250d8ae0f-kube-api-access-khfvk\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-image-registry-private-configuration\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439790 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66bca9b3-d1e5-475b-80a2-824250d8ae0f-config\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-registry-certificates\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-installation-pull-secrets\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.439915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4sk\" (UniqueName: \"kubernetes.io/projected/a9284783-b78b-4612-96b7-38fb17d6e4b4-kube-api-access-zw4sk\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9284783-b78b-4612-96b7-38fb17d6e4b4-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.443125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-trusted-ca\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.440348 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:30.940331568 +0000 UTC m=+37.629260805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : configmap references non-existent config key: service-ca.crt Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.440512 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzlq\" (UniqueName: \"kubernetes.io/projected/94d25ba2-c533-44bd-be10-c9da17f3929c-kube-api-access-srzlq\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-default-certificate\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d25ba2-c533-44bd-be10-c9da17f3929c-tmp-dir\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-nginx-conf\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vkdd\" (UniqueName: \"kubernetes.io/projected/65f05d41-05bb-4ac5-be33-f952fcbebdcb-kube-api-access-9vkdd\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mzn\" (UniqueName: \"kubernetes.io/projected/5a3562bb-92a7-4cc1-91bf-9df604d84f92-kube-api-access-w6mzn\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440514 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94d25ba2-c533-44bd-be10-c9da17f3929c-config-volume\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.440978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskw4\" (UniqueName: \"kubernetes.io/projected/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-kube-api-access-cskw4\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.443775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.441487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-trusted-ca\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.441643 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:30.941617564 +0000 UTC m=+37.630546798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : secret "router-metrics-certs-default" not found Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.441671 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.441727 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls podName:94d25ba2-c533-44bd-be10-c9da17f3929c nodeName:}" failed. No retries permitted until 2026-07-09 14:14:30.941711959 +0000 UTC m=+37.630641179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls") pod "dns-default-b8fdl" (UID: "94d25ba2-c533-44bd-be10-c9da17f3929c") : secret "dns-default-metrics-tls" not found Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.441953 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.441967 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c47c5d5b5-pnpfj: secret "image-registry-tls" not found Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.442098 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls podName:e8d1a01d-bbcc-490c-a195-09158937207f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:30.942084607 +0000 UTC m=+37.631013830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls") pod "image-registry-c47c5d5b5-pnpfj" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f") : secret "image-registry-tls" not found Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.442132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3506c053-ea2a-4484-b5c3-369c03d8baed-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.442186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-stats-auth\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.442218 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5a3562bb-92a7-4cc1-91bf-9df604d84f92-snapshots\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.442485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66bca9b3-d1e5-475b-80a2-824250d8ae0f-trusted-ca\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.442515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d1a01d-bbcc-490c-a195-09158937207f-ca-trust-extracted\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.442632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3562bb-92a7-4cc1-91bf-9df604d84f92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.444414 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.443333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d25ba2-c533-44bd-be10-c9da17f3929c-tmp-dir\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.443388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3506c053-ea2a-4484-b5c3-369c03d8baed-config\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.443438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-ca-trust-extracted\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.443446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3562bb-92a7-4cc1-91bf-9df604d84f92-tmp\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.443467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66bca9b3-d1e5-475b-80a2-824250d8ae0f-serving-cert\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.443502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-bound-sa-token\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.443525 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.443568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hbw\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-kube-api-access-d8hbw\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.444240 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3506c053-ea2a-4484-b5c3-369c03d8baed-config\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.444302 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls podName:a9284783-b78b-4612-96b7-38fb17d6e4b4 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:30.944286985 +0000 UTC m=+37.633216216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-24tn7" (UID: "a9284783-b78b-4612-96b7-38fb17d6e4b4") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.444344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-certificates\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.444403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrscx\" (UniqueName: \"kubernetes.io/projected/7e281907-2c2d-4944-8736-22f20ce41bb7-kube-api-access-wrscx\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.444853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-image-registry-private-configuration\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.445071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.445021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-default-certificate\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.445734 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.445100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:30.445734 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.445413 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a3562bb-92a7-4cc1-91bf-9df604d84f92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.446063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.446009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a3562bb-92a7-4cc1-91bf-9df604d84f92-serving-cert\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.446162 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.446142 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9284783-b78b-4612-96b7-38fb17d6e4b4-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.446613 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.446552 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3506c053-ea2a-4484-b5c3-369c03d8baed-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.446746 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.446726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-stats-auth\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.448158 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.448101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.448426 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.448348 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66bca9b3-d1e5-475b-80a2-824250d8ae0f-serving-cert\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.448612 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.448588 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-installation-pull-secrets\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.451525 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.451503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-bound-sa-token\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.452749 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.452724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sk42\" (UniqueName: \"kubernetes.io/projected/3506c053-ea2a-4484-b5c3-369c03d8baed-kube-api-access-5sk42\") pod \"service-ca-operator-7695fc6c8c-qnm92\" (UID: \"3506c053-ea2a-4484-b5c3-369c03d8baed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.454020 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.454000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfvk\" (UniqueName: \"kubernetes.io/projected/66bca9b3-d1e5-475b-80a2-824250d8ae0f-kube-api-access-khfvk\") pod \"console-operator-7866497494-wxw4j\" (UID: \"66bca9b3-d1e5-475b-80a2-824250d8ae0f\") " pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.454150 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.454131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4sk\" (UniqueName: \"kubernetes.io/projected/a9284783-b78b-4612-96b7-38fb17d6e4b4-kube-api-access-zw4sk\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.454560 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.454519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh4nj\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-kube-api-access-gh4nj\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.456081 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.456037 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrscx\" (UniqueName: \"kubernetes.io/projected/7e281907-2c2d-4944-8736-22f20ce41bb7-kube-api-access-wrscx\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.456669 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.456627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzlq\" (UniqueName: \"kubernetes.io/projected/94d25ba2-c533-44bd-be10-c9da17f3929c-kube-api-access-srzlq\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.457286 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.457198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnx7j\" (UniqueName: \"kubernetes.io/projected/7f992406-0c0a-420d-872d-b3a3dd9448a5-kube-api-access-dnx7j\") pod \"volume-data-source-validator-78495597f6-pkfdk\" (UID: \"7f992406-0c0a-420d-872d-b3a3dd9448a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" Jul 09 14:14:30.457523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.457501 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mzn\" (UniqueName: \"kubernetes.io/projected/5a3562bb-92a7-4cc1-91bf-9df604d84f92-kube-api-access-w6mzn\") pod \"insights-operator-68d9dd94cf-qptsb\" (UID: \"5a3562bb-92a7-4cc1-91bf-9df604d84f92\") " pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.458072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.458055 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskw4\" (UniqueName: \"kubernetes.io/projected/7d6410bb-ed3b-480e-bd9f-1bdef3e5e743-kube-api-access-cskw4\") pod \"kube-storage-version-migrator-operator-799cb7f958-8jz5b\" (UID: \"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.512405 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.512373 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" Jul 09 14:14:30.529400 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.529366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" Jul 09 14:14:30.546303 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-trusted-ca\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.546441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:30.546441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1055a6aa-997c-447b-ab11-452a2639dbd8-cert\") pod \"ingress-canary-sglr8\" (UID: \"1055a6aa-997c-447b-ab11-452a2639dbd8\") " pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.546441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqrd\" (UniqueName: \"kubernetes.io/projected/07d2bb68-32a6-4a7e-a1a7-21ad23767f8c-kube-api-access-lbqrd\") pod \"network-check-source-869dcfcdcd-27tzm\" (UID: \"07d2bb68-32a6-4a7e-a1a7-21ad23767f8c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" Jul 09 14:14:30.546599 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-image-registry-private-configuration\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.546599 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-installation-pull-secrets\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.546599 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-nginx-conf\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:30.546599 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.546599 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vkdd\" (UniqueName: \"kubernetes.io/projected/65f05d41-05bb-4ac5-be33-f952fcbebdcb-kube-api-access-9vkdd\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:30.546832 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-ca-trust-extracted\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.546832 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-bound-sa-token\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.546832 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hbw\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-kube-api-access-d8hbw\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.546832 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.546675 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:14:30.547024 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.546852 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert podName:967aa6b7-9ffe-405d-93c6-fdbc4c9a3715 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.046821719 +0000 UTC m=+37.735750956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-866jh" (UID: "967aa6b7-9ffe-405d-93c6-fdbc4c9a3715") : secret "networking-console-plugin-cert" not found Jul 09 14:14:30.547086 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.547040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-certificates\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.547528 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.546679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-certificates\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.547631 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.547589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:30.547686 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.547645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-ca-trust-extracted\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.547686 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.547653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv62d\" (UniqueName: \"kubernetes.io/projected/1055a6aa-997c-447b-ab11-452a2639dbd8-kube-api-access-sv62d\") pod \"ingress-canary-sglr8\" (UID: \"1055a6aa-997c-447b-ab11-452a2639dbd8\") " pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.547930 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.547799 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:30.547930 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.547812 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547b9bc97d-r5mw4: secret "image-registry-tls" not found Jul 09 14:14:30.547930 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.547877 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls podName:c4f86062-0ebd-4395-88d2-cca4bf6dd7dd nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.047858485 +0000 UTC m=+37.736787704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls") pod "image-registry-547b9bc97d-r5mw4" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd") : secret "image-registry-tls" not found Jul 09 14:14:30.548139 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.547949 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:14:30.548139 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.548001 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls podName:65f05d41-05bb-4ac5-be33-f952fcbebdcb nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.047984606 +0000 UTC m=+37.736913838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls") pod "cluster-samples-operator-65db858894-q8m2r" (UID: "65f05d41-05bb-4ac5-be33-f952fcbebdcb") : secret "samples-operator-tls" not found Jul 09 14:14:30.548474 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.548448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-nginx-conf\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:30.548911 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.548886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-trusted-ca\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.550110 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.550091 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1055a6aa-997c-447b-ab11-452a2639dbd8-cert\") pod \"ingress-canary-sglr8\" (UID: \"1055a6aa-997c-447b-ab11-452a2639dbd8\") " pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.550193 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.550132 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-image-registry-private-configuration\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.550653 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.550628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-installation-pull-secrets\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.552580 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.552540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:30.563003 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.562129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" Jul 09 14:14:30.563003 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.562907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqrd\" (UniqueName: \"kubernetes.io/projected/07d2bb68-32a6-4a7e-a1a7-21ad23767f8c-kube-api-access-lbqrd\") pod \"network-check-source-869dcfcdcd-27tzm\" (UID: \"07d2bb68-32a6-4a7e-a1a7-21ad23767f8c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" Jul 09 14:14:30.563557 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.563507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv62d\" (UniqueName: \"kubernetes.io/projected/1055a6aa-997c-447b-ab11-452a2639dbd8-kube-api-access-sv62d\") pod \"ingress-canary-sglr8\" (UID: \"1055a6aa-997c-447b-ab11-452a2639dbd8\") " pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.564067 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.564015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hbw\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-kube-api-access-d8hbw\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.564402 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.564381 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vkdd\" (UniqueName: \"kubernetes.io/projected/65f05d41-05bb-4ac5-be33-f952fcbebdcb-kube-api-access-9vkdd\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:30.564402 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.564411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-bound-sa-token\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:30.593242 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.592915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" Jul 09 14:14:30.596100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.595961 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" Jul 09 14:14:30.619167 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.618970 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sglr8" Jul 09 14:14:30.744769 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.744654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk"] Jul 09 14:14:30.764515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.764019 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92"] Jul 09 14:14:30.781722 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.780201 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-wxw4j"] Jul 09 14:14:30.790981 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.790889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qptsb"] Jul 09 14:14:30.795990 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:14:30.794918 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3562bb_92a7_4cc1_91bf_9df604d84f92.slice/crio-eb149b8e40d3880178148f5767b9c23c28e9f5c7eaae1c9903c27f1a01d27128 WatchSource:0}: Error finding container eb149b8e40d3880178148f5767b9c23c28e9f5c7eaae1c9903c27f1a01d27128: Status 404 returned error can't find the container with id eb149b8e40d3880178148f5767b9c23c28e9f5c7eaae1c9903c27f1a01d27128 Jul 09 14:14:30.832806 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.832747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm"] Jul 09 14:14:30.836178 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:14:30.836114 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d2bb68_32a6_4a7e_a1a7_21ad23767f8c.slice/crio-1067bbea753df7cc618c379ab19aabef9d7eb5e3287d2af33862dcac590ac8a5 WatchSource:0}: Error finding container 1067bbea753df7cc618c379ab19aabef9d7eb5e3287d2af33862dcac590ac8a5: Status 404 returned error can't find the container with id 1067bbea753df7cc618c379ab19aabef9d7eb5e3287d2af33862dcac590ac8a5 Jul 09 14:14:30.872412 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.872381 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b"] Jul 09 14:14:30.874613 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:14:30.874534 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6410bb_ed3b_480e_bd9f_1bdef3e5e743.slice/crio-c31bbbcc85ede3145017061a8a2bf97d076be3960f578c0f231ca6465f86997f WatchSource:0}: Error finding container c31bbbcc85ede3145017061a8a2bf97d076be3960f578c0f231ca6465f86997f: Status 404 returned error can't find the container with id c31bbbcc85ede3145017061a8a2bf97d076be3960f578c0f231ca6465f86997f Jul 09 14:14:30.875858 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.875837 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sglr8"] Jul 09 14:14:30.878286 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:14:30.878257 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1055a6aa_997c_447b_ab11_452a2639dbd8.slice/crio-311a3c295206e076b0ba364ba389a947b3a362ff33dddd95ed2d421842e1c349 WatchSource:0}: Error finding container 311a3c295206e076b0ba364ba389a947b3a362ff33dddd95ed2d421842e1c349: Status 404 returned error can't find the container with id 311a3c295206e076b0ba364ba389a947b3a362ff33dddd95ed2d421842e1c349 Jul 09 14:14:30.879889 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.879869 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:30.879977 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.879894 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:30.879977 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.879909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:30.882549 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.882155 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 14:14:30.882549 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.882284 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wzd4v\"" Jul 09 14:14:30.882549 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.882306 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cs7kw\"" Jul 09 14:14:30.882549 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.882425 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 14:14:30.954759 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.954718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.954960 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.954766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:30.954960 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.954888 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:14:30.954960 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.954903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:30.954960 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.954932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:30.954960 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.954959 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.954937357 +0000 UTC m=+38.643866583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : configmap references non-existent config key: service-ca.crt Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.954985 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.954975411 +0000 UTC m=+38.643904628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : secret "router-metrics-certs-default" not found Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.955023 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.955078 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.955083 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls podName:94d25ba2-c533-44bd-be10-c9da17f3929c nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.955065328 +0000 UTC m=+38.643994553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls") pod "dns-default-b8fdl" (UID: "94d25ba2-c533-44bd-be10-c9da17f3929c") : secret "dns-default-metrics-tls" not found Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.955027 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.955111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls podName:a9284783-b78b-4612-96b7-38fb17d6e4b4 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.955101542 +0000 UTC m=+38.644030765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-24tn7" (UID: "a9284783-b78b-4612-96b7-38fb17d6e4b4") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.955115 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c47c5d5b5-pnpfj: secret "image-registry-tls" not found Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:30.955020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:30.955208 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:30.955166 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls podName:e8d1a01d-bbcc-490c-a195-09158937207f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:31.955150715 +0000 UTC m=+38.644079958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls") pod "image-registry-c47c5d5b5-pnpfj" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f") : secret "image-registry-tls" not found Jul 09 14:14:31.056396 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.056330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:31.056563 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.056467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:31.056563 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.056487 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:14:31.056563 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.056529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:31.056563 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.056553 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert podName:967aa6b7-9ffe-405d-93c6-fdbc4c9a3715 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:32.056537189 +0000 UTC m=+38.745466410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-866jh" (UID: "967aa6b7-9ffe-405d-93c6-fdbc4c9a3715") : secret "networking-console-plugin-cert" not found Jul 09 14:14:31.056775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.056606 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:14:31.056775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.056656 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls podName:65f05d41-05bb-4ac5-be33-f952fcbebdcb nodeName:}" failed. No retries permitted until 2026-07-09 14:14:32.056641049 +0000 UTC m=+38.745570283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls") pod "cluster-samples-operator-65db858894-q8m2r" (UID: "65f05d41-05bb-4ac5-be33-f952fcbebdcb") : secret "samples-operator-tls" not found Jul 09 14:14:31.056775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.056610 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:31.056775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.056677 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547b9bc97d-r5mw4: secret "image-registry-tls" not found Jul 09 14:14:31.056775 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.056699 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls podName:c4f86062-0ebd-4395-88d2-cca4bf6dd7dd nodeName:}" failed. No retries permitted until 2026-07-09 14:14:32.056692921 +0000 UTC m=+38.745622141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls") pod "image-registry-547b9bc97d-r5mw4" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd") : secret "image-registry-tls" not found Jul 09 14:14:31.301000 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.300918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:31.304721 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.304689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" event={"ID":"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743","Type":"ContainerStarted","Data":"c31bbbcc85ede3145017061a8a2bf97d076be3960f578c0f231ca6465f86997f"} Jul 09 14:14:31.305969 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.305944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" event={"ID":"3506c053-ea2a-4484-b5c3-369c03d8baed","Type":"ContainerStarted","Data":"baa7641559273346611f0ed9b8caa3c389430d8c50e7ddda80e209bb4261d82d"} Jul 09 14:14:31.307197 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.307137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" event={"ID":"5a3562bb-92a7-4cc1-91bf-9df604d84f92","Type":"ContainerStarted","Data":"eb149b8e40d3880178148f5767b9c23c28e9f5c7eaae1c9903c27f1a01d27128"} Jul 09 14:14:31.308368 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.308342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wxw4j" event={"ID":"66bca9b3-d1e5-475b-80a2-824250d8ae0f","Type":"ContainerStarted","Data":"3c7212fa7a0cd5db53ab25b06e9601f6f4185d57f4df52a646c71996e8332876"} Jul 09 14:14:31.309604 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.309582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" event={"ID":"7f992406-0c0a-420d-872d-b3a3dd9448a5","Type":"ContainerStarted","Data":"fb808c1930ed04c159e9948cfed602c2714523f9ac9fe6da9a470122fe776fd6"} Jul 09 14:14:31.310851 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.310809 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sglr8" event={"ID":"1055a6aa-997c-447b-ab11-452a2639dbd8","Type":"ContainerStarted","Data":"311a3c295206e076b0ba364ba389a947b3a362ff33dddd95ed2d421842e1c349"} Jul 09 14:14:31.312026 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.312003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" event={"ID":"07d2bb68-32a6-4a7e-a1a7-21ad23767f8c","Type":"ContainerStarted","Data":"1067bbea753df7cc618c379ab19aabef9d7eb5e3287d2af33862dcac590ac8a5"} Jul 09 14:14:31.964067 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.964020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:31.964290 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.964085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:31.964290 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964169 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:14:31.964290 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.964208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:31.964290 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964261 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:33.964241642 +0000 UTC m=+40.653170878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : secret "router-metrics-certs-default" not found Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964290 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:33.96427125 +0000 UTC m=+40.653200478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : configmap references non-existent config key: service-ca.crt Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964315 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.964323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964351 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls podName:94d25ba2-c533-44bd-be10-c9da17f3929c nodeName:}" failed. No retries permitted until 2026-07-09 14:14:33.964340154 +0000 UTC m=+40.653269372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls") pod "dns-default-b8fdl" (UID: "94d25ba2-c533-44bd-be10-c9da17f3929c") : secret "dns-default-metrics-tls" not found Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:31.964368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964408 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964419 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c47c5d5b5-pnpfj: secret "image-registry-tls" not found Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964452 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls podName:e8d1a01d-bbcc-490c-a195-09158937207f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:33.964440843 +0000 UTC m=+40.653370062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls") pod "image-registry-c47c5d5b5-pnpfj" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f") : secret "image-registry-tls" not found Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964467 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:31.964543 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:31.964493 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls podName:a9284783-b78b-4612-96b7-38fb17d6e4b4 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:33.964486396 +0000 UTC m=+40.653415613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-24tn7" (UID: "a9284783-b78b-4612-96b7-38fb17d6e4b4") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:32.065232 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:32.065180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:32.065418 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:32.065270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:32.065418 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:32.065304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:32.065418 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:32.065347 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:32.065418 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:32.065384 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547b9bc97d-r5mw4: secret "image-registry-tls" not found Jul 09 14:14:32.065418 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:32.065417 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:14:32.065667 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:32.065417 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:14:32.065667 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:32.065471 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert podName:967aa6b7-9ffe-405d-93c6-fdbc4c9a3715 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:34.0654546 +0000 UTC m=+40.754383821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-866jh" (UID: "967aa6b7-9ffe-405d-93c6-fdbc4c9a3715") : secret "networking-console-plugin-cert" not found Jul 09 14:14:32.065667 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:32.065488 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls podName:c4f86062-0ebd-4395-88d2-cca4bf6dd7dd nodeName:}" failed. No retries permitted until 2026-07-09 14:14:34.065479285 +0000 UTC m=+40.754408501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls") pod "image-registry-547b9bc97d-r5mw4" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd") : secret "image-registry-tls" not found Jul 09 14:14:32.065667 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:32.065501 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls podName:65f05d41-05bb-4ac5-be33-f952fcbebdcb nodeName:}" failed. No retries permitted until 2026-07-09 14:14:34.065494011 +0000 UTC m=+40.754423229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls") pod "cluster-samples-operator-65db858894-q8m2r" (UID: "65f05d41-05bb-4ac5-be33-f952fcbebdcb") : secret "samples-operator-tls" not found Jul 09 14:14:33.327913 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:33.327835 2575 generic.go:356] "Generic (PLEG): container finished" podID="2323e31a-ed38-40a0-a730-ef6ef0acc26a" containerID="19b6cded1e721b5d47f206dd732f59dd051ddd2f95a88412d14d98dca6681fed" exitCode=0 Jul 09 14:14:33.327913 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:33.327905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerDied","Data":"19b6cded1e721b5d47f206dd732f59dd051ddd2f95a88412d14d98dca6681fed"} Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:33.986244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:33.986312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:33.986349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:33.986434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:33.986466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.986724 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:37.986703429 +0000 UTC m=+44.675632651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : configmap references non-existent config key: service-ca.crt Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987155 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987210 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls podName:94d25ba2-c533-44bd-be10-c9da17f3929c nodeName:}" failed. No retries permitted until 2026-07-09 14:14:37.987193761 +0000 UTC m=+44.676122982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls") pod "dns-default-b8fdl" (UID: "94d25ba2-c533-44bd-be10-c9da17f3929c") : secret "dns-default-metrics-tls" not found Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987290 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987301 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c47c5d5b5-pnpfj: secret "image-registry-tls" not found Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987333 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls podName:e8d1a01d-bbcc-490c-a195-09158937207f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:37.987321886 +0000 UTC m=+44.676251107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls") pod "image-registry-c47c5d5b5-pnpfj" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f") : secret "image-registry-tls" not found Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987388 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:33.987542 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987427 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls podName:a9284783-b78b-4612-96b7-38fb17d6e4b4 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:37.987414884 +0000 UTC m=+44.676344105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-24tn7" (UID: "a9284783-b78b-4612-96b7-38fb17d6e4b4") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:33.988348 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987474 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:14:33.988348 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:33.987502 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:37.98749304 +0000 UTC m=+44.676422258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : secret "router-metrics-certs-default" not found Jul 09 14:14:34.087430 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:34.087393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:34.087584 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:34.087567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:34.087654 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:34.087636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:34.087783 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:34.087769 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:14:34.088430 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:34.087834 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls podName:65f05d41-05bb-4ac5-be33-f952fcbebdcb nodeName:}" failed. No retries permitted until 2026-07-09 14:14:38.087813749 +0000 UTC m=+44.776742974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls") pod "cluster-samples-operator-65db858894-q8m2r" (UID: "65f05d41-05bb-4ac5-be33-f952fcbebdcb") : secret "samples-operator-tls" not found Jul 09 14:14:34.088430 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:34.088237 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:14:34.088430 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:34.088290 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert podName:967aa6b7-9ffe-405d-93c6-fdbc4c9a3715 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:38.088274743 +0000 UTC m=+44.777203963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-866jh" (UID: "967aa6b7-9ffe-405d-93c6-fdbc4c9a3715") : secret "networking-console-plugin-cert" not found Jul 09 14:14:34.088430 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:34.088354 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:34.088430 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:34.088365 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547b9bc97d-r5mw4: secret "image-registry-tls" not found Jul 09 14:14:34.088430 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:34.088400 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls podName:c4f86062-0ebd-4395-88d2-cca4bf6dd7dd nodeName:}" failed. No retries permitted until 2026-07-09 14:14:38.088389193 +0000 UTC m=+44.777318420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls") pod "image-registry-547b9bc97d-r5mw4" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd") : secret "image-registry-tls" not found Jul 09 14:14:34.350341 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:34.350301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerStarted","Data":"09f233089db8eec77aa00206db1a3a9a1ad6e39b457a1425c0339c4fe9a9a359"} Jul 09 14:14:35.356206 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:35.356177 2575 generic.go:356] "Generic (PLEG): container finished" podID="2323e31a-ed38-40a0-a730-ef6ef0acc26a" containerID="09f233089db8eec77aa00206db1a3a9a1ad6e39b457a1425c0339c4fe9a9a359" exitCode=0 Jul 09 14:14:35.356604 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:35.356251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerDied","Data":"09f233089db8eec77aa00206db1a3a9a1ad6e39b457a1425c0339c4fe9a9a359"} Jul 09 14:14:37.581136 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:37.581084 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:38.028131 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.028086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:38.028329 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.028193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:38.028329 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.028214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:38.028329 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.028252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:38.028329 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028302 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.028283082 +0000 UTC m=+52.717212298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : configmap references non-existent config key: service-ca.crt Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028352 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028356 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028373 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c47c5d5b5-pnpfj: secret "image-registry-tls" not found Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028356 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028416 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls podName:a9284783-b78b-4612-96b7-38fb17d6e4b4 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.028398701 +0000 UTC m=+52.717327944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-24tn7" (UID: "a9284783-b78b-4612-96b7-38fb17d6e4b4") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028439 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls podName:94d25ba2-c533-44bd-be10-c9da17f3929c nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.028427323 +0000 UTC m=+52.717356553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls") pod "dns-default-b8fdl" (UID: "94d25ba2-c533-44bd-be10-c9da17f3929c") : secret "dns-default-metrics-tls" not found Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028455 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls podName:e8d1a01d-bbcc-490c-a195-09158937207f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.028446528 +0000 UTC m=+52.717375745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls") pod "image-registry-c47c5d5b5-pnpfj" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f") : secret "image-registry-tls" not found Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.028436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:38.028495 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028490 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:14:38.028758 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.028519 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.028510509 +0000 UTC m=+52.717439729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : secret "router-metrics-certs-default" not found Jul 09 14:14:38.129122 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.129082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:38.129317 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.129197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:38.129317 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:38.129256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:38.129317 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.129295 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:14:38.129460 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.129314 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:38.129460 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.129326 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:14:38.129460 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.129333 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547b9bc97d-r5mw4: secret "image-registry-tls" not found Jul 09 14:14:38.129460 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.129375 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls podName:65f05d41-05bb-4ac5-be33-f952fcbebdcb nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.12936136 +0000 UTC m=+52.818290577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls") pod "cluster-samples-operator-65db858894-q8m2r" (UID: "65f05d41-05bb-4ac5-be33-f952fcbebdcb") : secret "samples-operator-tls" not found Jul 09 14:14:38.129460 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.129387 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert podName:967aa6b7-9ffe-405d-93c6-fdbc4c9a3715 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.129381896 +0000 UTC m=+52.818311113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-866jh" (UID: "967aa6b7-9ffe-405d-93c6-fdbc4c9a3715") : secret "networking-console-plugin-cert" not found Jul 09 14:14:38.129460 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:38.129417 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls podName:c4f86062-0ebd-4395-88d2-cca4bf6dd7dd nodeName:}" failed. No retries permitted until 2026-07-09 14:14:46.129411306 +0000 UTC m=+52.818340523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls") pod "image-registry-547b9bc97d-r5mw4" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd") : secret "image-registry-tls" not found Jul 09 14:14:40.092190 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.091811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.179023 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.178986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.255125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.255025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.336079 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.336046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.378387 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.378344 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" event={"ID":"5a3562bb-92a7-4cc1-91bf-9df604d84f92","Type":"ContainerStarted","Data":"cb232800c7088617e361280e888a426d754c2f04d108a834f58372956d355265"} Jul 09 14:14:40.380043 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.380019 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/0.log" Jul 09 14:14:40.380182 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.380059 2575 generic.go:356] "Generic (PLEG): container finished" podID="66bca9b3-d1e5-475b-80a2-824250d8ae0f" containerID="3b15d60c84017f0f100289ccbf642dad0849fc03a65882805095fcc31ba2a4b0" exitCode=255 Jul 09 14:14:40.380182 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.380140 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wxw4j" event={"ID":"66bca9b3-d1e5-475b-80a2-824250d8ae0f","Type":"ContainerDied","Data":"3b15d60c84017f0f100289ccbf642dad0849fc03a65882805095fcc31ba2a4b0"} Jul 09 14:14:40.380370 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.380353 2575 scope.go:117] "RemoveContainer" containerID="3b15d60c84017f0f100289ccbf642dad0849fc03a65882805095fcc31ba2a4b0" Jul 09 14:14:40.388169 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.388149 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" event={"ID":"2323e31a-ed38-40a0-a730-ef6ef0acc26a","Type":"ContainerStarted","Data":"2bb2d47c3fd3ad9695b4143e2614145b2227e63db9d4f6d5fbd76dddae969ee7"} Jul 09 14:14:40.389758 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.389734 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" event={"ID":"7f992406-0c0a-420d-872d-b3a3dd9448a5","Type":"ContainerStarted","Data":"7936ed13e7853b506c50919af800eb332ef4efb0c5e3107565afcba08d2d83eb"} Jul 09 14:14:40.391248 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.391206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sglr8" event={"ID":"1055a6aa-997c-447b-ab11-452a2639dbd8","Type":"ContainerStarted","Data":"f5ef3acf6e4ab0f52dd6a23ce974cc6f8a5766e3367adf2046d897d7debe236a"} Jul 09 14:14:40.392745 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.392725 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" event={"ID":"07d2bb68-32a6-4a7e-a1a7-21ad23767f8c","Type":"ContainerStarted","Data":"62b661b59b24d44686de2c26989bd6f003fa38f114ae94f4fd110ed15e6c159e"} Jul 09 14:14:40.394335 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.394316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" event={"ID":"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743","Type":"ContainerStarted","Data":"04990c7141f0e630954e679f62c8a636f9f22020700221b5c3fb06ae181f4989"} Jul 09 14:14:40.395815 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.395796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" event={"ID":"3506c053-ea2a-4484-b5c3-369c03d8baed","Type":"ContainerStarted","Data":"9c9b2be2511a67cb2d3fa67c7161d98fa4ae9cc5332ccf9b311c7573744e88b5"} Jul 09 14:14:40.398902 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.398865 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" podStartSLOduration=10.577643662 podStartE2EDuration="19.398845068s" totalImagesPullingTime="8.821201406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:40.397615194 +0000 UTC m=+47.086544434" watchObservedRunningTime="2026-07-09 14:14:40.398845068 +0000 UTC m=+47.087774306" Jul 09 14:14:40.404603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.404565 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.419992 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.419930 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sglr8" podStartSLOduration=1.645417767 podStartE2EDuration="10.41991209s" totalImagesPullingTime="8.774494323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:40.419825072 +0000 UTC m=+47.108754335" watchObservedRunningTime="2026-07-09 14:14:40.41991209 +0000 UTC m=+47.108841330" Jul 09 14:14:40.460382 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.460325 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-27tzm" podStartSLOduration=7.629528442 podStartE2EDuration="16.460304342s" totalImagesPullingTime="8.8307759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:40.459207748 +0000 UTC m=+47.148137003" watchObservedRunningTime="2026-07-09 14:14:40.460304342 +0000 UTC m=+47.149233584" Jul 09 14:14:40.461099 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.461058 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" podStartSLOduration=10.682605564 podStartE2EDuration="19.461038831s" totalImagesPullingTime="8.778433267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:40.441043613 +0000 UTC m=+47.129972853" watchObservedRunningTime="2026-07-09 14:14:40.461038831 +0000 UTC m=+47.149968070" Jul 09 14:14:40.489839 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.489790 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mwrvj" podStartSLOduration=15.508864721 podStartE2EDuration="45.48977546s" totalImagesPullingTime="29.980910739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:40.489386776 +0000 UTC m=+47.178316056" watchObservedRunningTime="2026-07-09 14:14:40.48977546 +0000 UTC m=+47.178704699" Jul 09 14:14:40.510901 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.509822 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-pkfdk" podStartSLOduration=12.219706303 podStartE2EDuration="19.509801791s" totalImagesPullingTime="7.290095488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:40.509317248 +0000 UTC m=+47.198246514" watchObservedRunningTime="2026-07-09 14:14:40.509801791 +0000 UTC m=+47.198731033" Jul 09 14:14:40.556393 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.554989 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" podStartSLOduration=10.668492285 podStartE2EDuration="19.554965478s" totalImagesPullingTime="8.886473193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:40.532746504 +0000 UTC m=+47.221675744" watchObservedRunningTime="2026-07-09 14:14:40.554965478 +0000 UTC m=+47.243894718" Jul 09 14:14:40.556393 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.556302 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:40.556393 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.556367 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:40.636110 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.636067 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.740309 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.740245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.887493 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.887453 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:40.984755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:40.984713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.075241 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.075186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.168615 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.168525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.347497 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.347467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.354568 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.354537 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv"] Jul 09 14:14:41.368741 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.368718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv"] Jul 09 14:14:41.368914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.368818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" Jul 09 14:14:41.374125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.373861 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:41.374125 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.373885 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 14:14:41.374683 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.374660 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sllng\"" Jul 09 14:14:41.401839 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.401813 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/1.log" Jul 09 14:14:41.402297 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.402281 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/0.log" Jul 09 14:14:41.402360 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.402321 2575 generic.go:356] "Generic (PLEG): container finished" podID="66bca9b3-d1e5-475b-80a2-824250d8ae0f" containerID="504a789522a95dfd889e21fd4a6b1ff1d712a8945565647019e9b30b197cb1e6" exitCode=255 Jul 09 14:14:41.402441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.402420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wxw4j" event={"ID":"66bca9b3-d1e5-475b-80a2-824250d8ae0f","Type":"ContainerDied","Data":"504a789522a95dfd889e21fd4a6b1ff1d712a8945565647019e9b30b197cb1e6"} Jul 09 14:14:41.402498 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.402484 2575 scope.go:117] "RemoveContainer" containerID="3b15d60c84017f0f100289ccbf642dad0849fc03a65882805095fcc31ba2a4b0" Jul 09 14:14:41.402630 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.402601 2575 scope.go:117] "RemoveContainer" containerID="504a789522a95dfd889e21fd4a6b1ff1d712a8945565647019e9b30b197cb1e6" Jul 09 14:14:41.402787 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:41.402764 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wxw4j_openshift-console-operator(66bca9b3-d1e5-475b-80a2-824250d8ae0f)\"" pod="openshift-console-operator/console-operator-7866497494-wxw4j" podUID="66bca9b3-d1e5-475b-80a2-824250d8ae0f" Jul 09 14:14:41.429467 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.429439 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.467606 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.467550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lgc\" (UniqueName: \"kubernetes.io/projected/7a0f826f-d4ab-4b28-bda1-ba9f9a674b65-kube-api-access-x2lgc\") pod \"migrator-7f59b54f6c-zzdzv\" (UID: \"7a0f826f-d4ab-4b28-bda1-ba9f9a674b65\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" Jul 09 14:14:41.497489 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.497444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.568868 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.568828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lgc\" (UniqueName: \"kubernetes.io/projected/7a0f826f-d4ab-4b28-bda1-ba9f9a674b65-kube-api-access-x2lgc\") pod \"migrator-7f59b54f6c-zzdzv\" (UID: \"7a0f826f-d4ab-4b28-bda1-ba9f9a674b65\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" Jul 09 14:14:41.581138 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.581095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lgc\" (UniqueName: \"kubernetes.io/projected/7a0f826f-d4ab-4b28-bda1-ba9f9a674b65-kube-api-access-x2lgc\") pod \"migrator-7f59b54f6c-zzdzv\" (UID: \"7a0f826f-d4ab-4b28-bda1-ba9f9a674b65\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" Jul 09 14:14:41.599776 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.599738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.636283 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.636247 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s2gbj"] Jul 09 14:14:41.652944 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.652894 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2gbj"] Jul 09 14:14:41.653122 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.653010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.655693 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.655669 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 14:14:41.655848 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.655781 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 14:14:41.655929 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.655912 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dj2q5\"" Jul 09 14:14:41.676206 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.676127 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.680905 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.680883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" Jul 09 14:14:41.742965 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.742763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.771924 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.771847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-crio-socket\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.772031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.772013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-data-volume\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.772100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.772048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.772192 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.772177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.772358 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.772341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krhsh\" (UniqueName: \"kubernetes.io/projected/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-api-access-krhsh\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.807094 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.807060 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:41.818379 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.818353 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv"] Jul 09 14:14:41.822788 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:14:41.822757 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0f826f_d4ab_4b28_bda1_ba9f9a674b65.slice/crio-859ab414f3b9727bc2a274390d9663d626be05c91987cfaa2cfd6421827c5190 WatchSource:0}: Error finding container 859ab414f3b9727bc2a274390d9663d626be05c91987cfaa2cfd6421827c5190: Status 404 returned error can't find the container with id 859ab414f3b9727bc2a274390d9663d626be05c91987cfaa2cfd6421827c5190 Jul 09 14:14:41.873452 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.873417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-crio-socket\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.873629 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.873485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-data-volume\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.873629 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.873510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.873629 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.873553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.873629 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.873626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krhsh\" (UniqueName: \"kubernetes.io/projected/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-api-access-krhsh\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.873848 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.873686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-crio-socket\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.873848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:41.873695 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:14:41.873848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:41.873734 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:14:41.873848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:41.873805 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:42.373786282 +0000 UTC m=+49.062715522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : configmap "kube-rbac-proxy" not found Jul 09 14:14:41.873848 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:41.873826 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:42.373817004 +0000 UTC m=+49.062746222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : secret "insights-runtime-extractor-tls" not found Jul 09 14:14:41.874123 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.873961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-data-volume\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:41.883238 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:41.883201 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krhsh\" (UniqueName: \"kubernetes.io/projected/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-api-access-krhsh\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:42.021597 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.021555 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:42.122621 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.122588 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:42.326715 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.326633 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:42.378799 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.378761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:42.378961 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.378837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:42.378961 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:42.378925 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:14:42.378961 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:42.378953 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:14:42.379099 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:42.378986 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:43.378971394 +0000 UTC m=+50.067900611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : secret "insights-runtime-extractor-tls" not found Jul 09 14:14:42.379099 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:42.379008 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:43.379001229 +0000 UTC m=+50.067930446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : configmap "kube-rbac-proxy" not found Jul 09 14:14:42.409569 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.409545 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/1.log" Jul 09 14:14:42.409943 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.409925 2575 scope.go:117] "RemoveContainer" containerID="504a789522a95dfd889e21fd4a6b1ff1d712a8945565647019e9b30b197cb1e6" Jul 09 14:14:42.410160 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:42.410141 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wxw4j_openshift-console-operator(66bca9b3-d1e5-475b-80a2-824250d8ae0f)\"" pod="openshift-console-operator/console-operator-7866497494-wxw4j" podUID="66bca9b3-d1e5-475b-80a2-824250d8ae0f" Jul 09 14:14:42.410771 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.410749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" event={"ID":"7a0f826f-d4ab-4b28-bda1-ba9f9a674b65","Type":"ContainerStarted","Data":"859ab414f3b9727bc2a274390d9663d626be05c91987cfaa2cfd6421827c5190"} Jul 09 14:14:42.493117 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.493089 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:42.712696 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.712611 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:42.892409 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:42.892373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:43.273195 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.273160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:43.356535 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.356497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:43.389500 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.389474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:43.389650 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.389560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:43.389650 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:43.389619 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:14:43.389718 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:43.389684 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:14:43.389753 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:43.389721 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:45.389700065 +0000 UTC m=+52.078629282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : secret "insights-runtime-extractor-tls" not found Jul 09 14:14:43.389753 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:43.389741 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:45.389731062 +0000 UTC m=+52.078660283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : configmap "kube-rbac-proxy" not found Jul 09 14:14:43.417440 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.417419 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fdr6_aad276ab-66e3-4cc8-a0a5-6749b625419b/dns-node-resolver/0.log" Jul 09 14:14:43.608577 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.608504 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2mznj"] Jul 09 14:14:43.620091 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.620063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.622259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.622188 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2mznj"] Jul 09 14:14:43.622661 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.622645 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 14:14:43.622661 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.622656 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 14:14:43.622809 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.622657 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-flkzt\"" Jul 09 14:14:43.623463 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.623442 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 14:14:43.623553 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.623482 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 14:14:43.655504 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.655468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:43.691826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.691794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14b698d6-1c22-43ed-b6b0-6f58eae18131-signing-cabundle\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.692027 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.691857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14b698d6-1c22-43ed-b6b0-6f58eae18131-signing-key\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.692089 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.692034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhj6\" (UniqueName: \"kubernetes.io/projected/14b698d6-1c22-43ed-b6b0-6f58eae18131-kube-api-access-6xhj6\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.793004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.792967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhj6\" (UniqueName: \"kubernetes.io/projected/14b698d6-1c22-43ed-b6b0-6f58eae18131-kube-api-access-6xhj6\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.793186 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.793164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14b698d6-1c22-43ed-b6b0-6f58eae18131-signing-cabundle\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.793291 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.793217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14b698d6-1c22-43ed-b6b0-6f58eae18131-signing-key\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.793923 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.793899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14b698d6-1c22-43ed-b6b0-6f58eae18131-signing-cabundle\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.795862 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.795836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14b698d6-1c22-43ed-b6b0-6f58eae18131-signing-key\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.802502 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.802480 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhj6\" (UniqueName: \"kubernetes.io/projected/14b698d6-1c22-43ed-b6b0-6f58eae18131-kube-api-access-6xhj6\") pod \"service-ca-79f68b9f4f-2mznj\" (UID: \"14b698d6-1c22-43ed-b6b0-6f58eae18131\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:43.931418 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:43.931341 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" Jul 09 14:14:44.077528 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:44.077495 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2mznj"] Jul 09 14:14:44.097793 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:14:44.097766 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b698d6_1c22_43ed_b6b0_6f58eae18131.slice/crio-73575998d21476cca185246a8da7a56f3af557bb804203967c74f90bc722eb65 WatchSource:0}: Error finding container 73575998d21476cca185246a8da7a56f3af557bb804203967c74f90bc722eb65: Status 404 returned error can't find the container with id 73575998d21476cca185246a8da7a56f3af557bb804203967c74f90bc722eb65 Jul 09 14:14:44.418192 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:44.418165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" event={"ID":"7a0f826f-d4ab-4b28-bda1-ba9f9a674b65","Type":"ContainerStarted","Data":"d36de25aebc990b8a10683d1865b96b608277f8fffef97576b71d92690a03699"} Jul 09 14:14:44.419717 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:44.419691 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" event={"ID":"14b698d6-1c22-43ed-b6b0-6f58eae18131","Type":"ContainerStarted","Data":"6c915218f48beb4fbba02f33cb294cc6a92d2ee61929af0cd27eeae399544845"} Jul 09 14:14:44.419825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:44.419739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" event={"ID":"14b698d6-1c22-43ed-b6b0-6f58eae18131","Type":"ContainerStarted","Data":"73575998d21476cca185246a8da7a56f3af557bb804203967c74f90bc722eb65"} Jul 09 14:14:44.424423 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:44.424406 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9cbwm_0115aff2-b98a-4415-8204-4de41320eb9a/node-ca/0.log" Jul 09 14:14:44.441163 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:44.441124 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-2mznj" podStartSLOduration=1.4411094549999999 podStartE2EDuration="1.441109455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:44.440282653 +0000 UTC m=+51.129211918" watchObservedRunningTime="2026-07-09 14:14:44.441109455 +0000 UTC m=+51.130038724" Jul 09 14:14:44.716868 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:44.716831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:45.217602 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.217571 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sglr8_1055a6aa-997c-447b-ab11-452a2639dbd8/serve-healthcheck-canary/0.log" Jul 09 14:14:45.408994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.408961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:45.409153 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.409028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:45.409153 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:45.409112 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:14:45.409254 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:45.409171 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:14:45.409254 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:45.409178 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:49.409164151 +0000 UTC m=+56.098093380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : secret "insights-runtime-extractor-tls" not found Jul 09 14:14:45.409331 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:45.409266 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:49.409246189 +0000 UTC m=+56.098175419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : configmap "kube-rbac-proxy" not found Jul 09 14:14:45.424046 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.424014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" event={"ID":"7a0f826f-d4ab-4b28-bda1-ba9f9a674b65","Type":"ContainerStarted","Data":"59540c8519e148a0245e63d0e31c7e9b3d5cbcc64eb6c859005d086cb3c02c03"} Jul 09 14:14:45.445290 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.445240 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-zzdzv" podStartSLOduration=2.009138063 podStartE2EDuration="4.445206532s" totalImagesPullingTime="2.436068469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:45.444142517 +0000 UTC m=+52.133071759" watchObservedRunningTime="2026-07-09 14:14:45.445206532 +0000 UTC m=+52.134135773" Jul 09 14:14:45.616181 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.616153 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-zzdzv_7a0f826f-d4ab-4b28-bda1-ba9f9a674b65/graceful-termination/0.log" Jul 09 14:14:45.796541 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.796503 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:45.818826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.818796 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-8jz5b_7d6410bb-ed3b-480e-bd9f-1bdef3e5e743/kube-storage-version-migrator-operator/0.log" Jul 09 14:14:45.895761 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.895668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:45.997900 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:45.997862 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:46.095831 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.095795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:46.118046 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.118013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:46.118263 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.118060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:14:46.118263 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118176 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:14:46.118263 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.118201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:14:46.118263 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118262 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:02.118241066 +0000 UTC m=+68.807170296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : secret "router-metrics-certs-default" not found Jul 09 14:14:46.118482 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.118287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:14:46.118482 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118325 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:14:46.118482 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.118330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:14:46.118482 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118373 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls podName:94d25ba2-c533-44bd-be10-c9da17f3929c nodeName:}" failed. No retries permitted until 2026-07-09 14:15:02.118358165 +0000 UTC m=+68.807287387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls") pod "dns-default-b8fdl" (UID: "94d25ba2-c533-44bd-be10-c9da17f3929c") : secret "dns-default-metrics-tls" not found Jul 09 14:14:46.118482 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118428 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:46.118482 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118452 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle podName:7e281907-2c2d-4944-8736-22f20ce41bb7 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:02.118441815 +0000 UTC m=+68.807371039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle") pod "router-default-74555698d-dkf2p" (UID: "7e281907-2c2d-4944-8736-22f20ce41bb7") : configmap references non-existent config key: service-ca.crt Jul 09 14:14:46.118482 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118469 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls podName:a9284783-b78b-4612-96b7-38fb17d6e4b4 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:02.11846058 +0000 UTC m=+68.807389800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-24tn7" (UID: "a9284783-b78b-4612-96b7-38fb17d6e4b4") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:14:46.118837 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118517 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:46.118837 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118527 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c47c5d5b5-pnpfj: secret "image-registry-tls" not found Jul 09 14:14:46.118837 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.118574 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls podName:e8d1a01d-bbcc-490c-a195-09158937207f nodeName:}" failed. No retries permitted until 2026-07-09 14:15:02.118547181 +0000 UTC m=+68.807476404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls") pod "image-registry-c47c5d5b5-pnpfj" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f") : secret "image-registry-tls" not found Jul 09 14:14:46.219013 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.218914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:14:46.219013 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.219007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:46.219263 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.219056 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:14:46.219263 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.219065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:14:46.219263 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.219076 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547b9bc97d-r5mw4: secret "image-registry-tls" not found Jul 09 14:14:46.219263 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.219128 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls podName:c4f86062-0ebd-4395-88d2-cca4bf6dd7dd nodeName:}" failed. No retries permitted until 2026-07-09 14:15:02.219113245 +0000 UTC m=+68.908042461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls") pod "image-registry-547b9bc97d-r5mw4" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd") : secret "image-registry-tls" not found Jul 09 14:14:46.219263 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.219179 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:14:46.219263 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:46.219250 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert podName:967aa6b7-9ffe-405d-93c6-fdbc4c9a3715 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:02.21921281 +0000 UTC m=+68.908142033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-866jh" (UID: "967aa6b7-9ffe-405d-93c6-fdbc4c9a3715") : secret "networking-console-plugin-cert" not found Jul 09 14:14:46.221627 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.221597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/65f05d41-05bb-4ac5-be33-f952fcbebdcb-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q8m2r\" (UID: \"65f05d41-05bb-4ac5-be33-f952fcbebdcb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:46.407777 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.407746 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:46.504308 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.504209 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" Jul 09 14:14:46.647557 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:46.647493 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r"] Jul 09 14:14:47.094881 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:47.094843 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:47.432141 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:47.432062 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" event={"ID":"65f05d41-05bb-4ac5-be33-f952fcbebdcb","Type":"ContainerStarted","Data":"1c3b292c215f7a4de3b6e552e8ace5e95285559e019dcc987d7ecbe0e2bb12e4"} Jul 09 14:14:47.746155 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:47.746066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:47.879388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:47.879349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:47.977640 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:47.977608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:48.086362 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:48.086319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:48.802804 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:48.802770 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:48.903340 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:48.903305 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:49.349343 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.349311 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:49.440931 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.440908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" event={"ID":"65f05d41-05bb-4ac5-be33-f952fcbebdcb","Type":"ContainerStarted","Data":"92077e5b4b2d01e44e2aeab6a7396771366cf4939d02fa8074a4edd88029de9e"} Jul 09 14:14:49.457621 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.457584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:49.457737 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.457659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:49.458113 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:49.457830 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:14:49.458113 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:49.457892 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:57.457873315 +0000 UTC m=+64.146802558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : configmap "kube-rbac-proxy" not found Jul 09 14:14:49.458113 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:49.457917 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:14:49.458113 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:49.457980 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls podName:14e7daa9-1b76-445a-9b5a-b3106f89ff9d nodeName:}" failed. No retries permitted until 2026-07-09 14:14:57.457961524 +0000 UTC m=+64.146890758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2gbj" (UID: "14e7daa9-1b76-445a-9b5a-b3106f89ff9d") : secret "insights-runtime-extractor-tls" not found Jul 09 14:14:49.486214 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.486187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:49.586985 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.586954 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:49.828254 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.828198 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:49.928925 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:49.928888 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:50.086499 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:50.086400 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:50.151060 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:50.151025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:50.445737 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:50.445653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" event={"ID":"65f05d41-05bb-4ac5-be33-f952fcbebdcb","Type":"ContainerStarted","Data":"2a08201e0b97eaf39279be57e17d125aea659462a284c4df62d1b8a07ced33ff"} Jul 09 14:14:50.464827 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:50.464772 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q8m2r" podStartSLOduration=26.82119558 podStartE2EDuration="29.464756787s" totalImagesPullingTime="2.643561207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:50.463542675 +0000 UTC m=+57.152471914" watchObservedRunningTime="2026-07-09 14:14:50.464756787 +0000 UTC m=+57.153686026" Jul 09 14:14:50.552783 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:50.552731 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:50.552925 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:50.552805 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:14:50.553186 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:50.553171 2575 scope.go:117] "RemoveContainer" containerID="504a789522a95dfd889e21fd4a6b1ff1d712a8945565647019e9b30b197cb1e6" Jul 09 14:14:51.075268 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.075215 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:51.450314 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.450216 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:14:51.450668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.450651 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/1.log" Jul 09 14:14:51.450728 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.450687 2575 generic.go:356] "Generic (PLEG): container finished" podID="66bca9b3-d1e5-475b-80a2-824250d8ae0f" containerID="765ed366daee50c5c17c44b321866fca19522fdd85f13585a9ebc1a43941967e" exitCode=255 Jul 09 14:14:51.450791 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.450768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wxw4j" event={"ID":"66bca9b3-d1e5-475b-80a2-824250d8ae0f","Type":"ContainerDied","Data":"765ed366daee50c5c17c44b321866fca19522fdd85f13585a9ebc1a43941967e"} Jul 09 14:14:51.450847 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.450834 2575 scope.go:117] "RemoveContainer" containerID="504a789522a95dfd889e21fd4a6b1ff1d712a8945565647019e9b30b197cb1e6" Jul 09 14:14:51.451198 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.451179 2575 scope.go:117] "RemoveContainer" containerID="765ed366daee50c5c17c44b321866fca19522fdd85f13585a9ebc1a43941967e" Jul 09 14:14:51.451387 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:14:51.451367 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-wxw4j_openshift-console-operator(66bca9b3-d1e5-475b-80a2-824250d8ae0f)\"" pod="openshift-console-operator/console-operator-7866497494-wxw4j" podUID="66bca9b3-d1e5-475b-80a2-824250d8ae0f" Jul 09 14:14:51.486291 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.486254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:51.732754 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.732659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:51.792211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.792169 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:51.877374 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:51.877340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:52.098838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:52.098803 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:52.363956 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:52.363870 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:52.455359 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:52.455334 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:14:52.883582 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:52.883550 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:55.301315 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:55.301288 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw9lp" Jul 09 14:14:57.540132 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:57.540069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:57.540655 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:57.540264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:57.540793 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:57.540769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:57.542467 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:57.542448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e7daa9-1b76-445a-9b5a-b3106f89ff9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2gbj\" (UID: \"14e7daa9-1b76-445a-9b5a-b3106f89ff9d\") " pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:57.566305 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:57.566281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dj2q5\"" Jul 09 14:14:57.574439 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:57.574416 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2gbj" Jul 09 14:14:57.700243 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:57.700056 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2gbj"] Jul 09 14:14:57.702713 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:14:57.702686 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e7daa9_1b76_445a_9b5a_b3106f89ff9d.slice/crio-4bb6410e10d4b5930acdc2903ac305d8a821d45067e85e156799999c78305a12 WatchSource:0}: Error finding container 4bb6410e10d4b5930acdc2903ac305d8a821d45067e85e156799999c78305a12: Status 404 returned error can't find the container with id 4bb6410e10d4b5930acdc2903ac305d8a821d45067e85e156799999c78305a12 Jul 09 14:14:58.480361 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:58.480327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2gbj" event={"ID":"14e7daa9-1b76-445a-9b5a-b3106f89ff9d","Type":"ContainerStarted","Data":"4bb6410e10d4b5930acdc2903ac305d8a821d45067e85e156799999c78305a12"} Jul 09 14:14:59.661031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.660716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:59.661031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.660784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:59.661031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.660883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:59.663855 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.663801 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tmd\" (UniqueName: \"kubernetes.io/projected/50732da1-a962-4c24-8c2e-f71c931e04c1-kube-api-access-r9tmd\") pod \"network-check-target-lfhcd\" (UID: \"50732da1-a962-4c24-8c2e-f71c931e04c1\") " pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:14:59.663997 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.663928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 14:14:59.664887 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.664863 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 14:14:59.673444 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.673421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2cdcb585-98f7-4c44-928e-68b3dc41164e-original-pull-secret\") pod \"global-pull-secret-syncer-bgmqg\" (UID: \"2cdcb585-98f7-4c44-928e-68b3dc41164e\") " pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:59.673575 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.673556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e810efc5-e567-4924-af70-18dd39817937-metrics-certs\") pod \"network-metrics-daemon-8jdld\" (UID: \"e810efc5-e567-4924-af70-18dd39817937\") " pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:59.692737 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.692701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgmqg" Jul 09 14:14:59.702544 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.702500 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cs7kw\"" Jul 09 14:14:59.709464 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.709439 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wzd4v\"" Jul 09 14:14:59.710449 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.710427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jdld" Jul 09 14:14:59.717886 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:14:59.717850 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:15:00.553140 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:00.553106 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:15:00.553325 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:00.553177 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:15:00.553605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:00.553589 2575 scope.go:117] "RemoveContainer" containerID="765ed366daee50c5c17c44b321866fca19522fdd85f13585a9ebc1a43941967e" Jul 09 14:15:00.553821 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:15:00.553783 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-wxw4j_openshift-console-operator(66bca9b3-d1e5-475b-80a2-824250d8ae0f)\"" pod="openshift-console-operator/console-operator-7866497494-wxw4j" podUID="66bca9b3-d1e5-475b-80a2-824250d8ae0f" Jul 09 14:15:01.411714 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.411686 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bgmqg"] Jul 09 14:15:01.415325 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:01.415292 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdcb585_98f7_4c44_928e_68b3dc41164e.slice/crio-7e4a579f1bb1aebc9bdd5630aa7a13538ea2374aecfbfd38216750ba474519e6 WatchSource:0}: Error finding container 7e4a579f1bb1aebc9bdd5630aa7a13538ea2374aecfbfd38216750ba474519e6: Status 404 returned error can't find the container with id 7e4a579f1bb1aebc9bdd5630aa7a13538ea2374aecfbfd38216750ba474519e6 Jul 09 14:15:01.431783 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.431761 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfhcd"] Jul 09 14:15:01.435053 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:01.435029 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50732da1_a962_4c24_8c2e_f71c931e04c1.slice/crio-8936097340f0a78276e9a14392177156f51c4f39d2765d908b5d7eef678ec8da WatchSource:0}: Error finding container 8936097340f0a78276e9a14392177156f51c4f39d2765d908b5d7eef678ec8da: Status 404 returned error can't find the container with id 8936097340f0a78276e9a14392177156f51c4f39d2765d908b5d7eef678ec8da Jul 09 14:15:01.449286 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.449254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jdld"] Jul 09 14:15:01.452113 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:01.452083 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode810efc5_e567_4924_af70_18dd39817937.slice/crio-5916eae9d86b56f91a099aec00e9d285899e640a54573100a7086173fbb4165b WatchSource:0}: Error finding container 5916eae9d86b56f91a099aec00e9d285899e640a54573100a7086173fbb4165b: Status 404 returned error can't find the container with id 5916eae9d86b56f91a099aec00e9d285899e640a54573100a7086173fbb4165b Jul 09 14:15:01.495846 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.495813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jdld" event={"ID":"e810efc5-e567-4924-af70-18dd39817937","Type":"ContainerStarted","Data":"5916eae9d86b56f91a099aec00e9d285899e640a54573100a7086173fbb4165b"} Jul 09 14:15:01.496864 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.496839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bgmqg" event={"ID":"2cdcb585-98f7-4c44-928e-68b3dc41164e","Type":"ContainerStarted","Data":"7e4a579f1bb1aebc9bdd5630aa7a13538ea2374aecfbfd38216750ba474519e6"} Jul 09 14:15:01.498380 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.498354 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfhcd" event={"ID":"50732da1-a962-4c24-8c2e-f71c931e04c1","Type":"ContainerStarted","Data":"55c29849aa6678646726007d0d16491c0546ab19ab80cc5320a3ebfffdf0c089"} Jul 09 14:15:01.498476 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.498396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfhcd" event={"ID":"50732da1-a962-4c24-8c2e-f71c931e04c1","Type":"ContainerStarted","Data":"8936097340f0a78276e9a14392177156f51c4f39d2765d908b5d7eef678ec8da"} Jul 09 14:15:01.498476 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.498460 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:15:01.499923 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.499895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2gbj" event={"ID":"14e7daa9-1b76-445a-9b5a-b3106f89ff9d","Type":"ContainerStarted","Data":"c1f5addd9020f7a186dabca5f1822db17f535196ae0adb5bc722575355a251a1"} Jul 09 14:15:01.517048 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:01.516994 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lfhcd" podStartSLOduration=66.516979636 podStartE2EDuration="1m6.516979636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:01.516035109 +0000 UTC m=+68.204964347" watchObservedRunningTime="2026-07-09 14:15:01.516979636 +0000 UTC m=+68.205908875" Jul 09 14:15:02.188998 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.188959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:15:02.188998 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.189006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:15:02.189203 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.189032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:15:02.189203 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.189079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:02.189203 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.189107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:02.189852 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.189828 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e281907-2c2d-4944-8736-22f20ce41bb7-service-ca-bundle\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:02.191701 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.191676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e281907-2c2d-4944-8736-22f20ce41bb7-metrics-certs\") pod \"router-default-74555698d-dkf2p\" (UID: \"7e281907-2c2d-4944-8736-22f20ce41bb7\") " pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:02.191794 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.191675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"image-registry-c47c5d5b5-pnpfj\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:15:02.191794 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.191745 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94d25ba2-c533-44bd-be10-c9da17f3929c-metrics-tls\") pod \"dns-default-b8fdl\" (UID: \"94d25ba2-c533-44bd-be10-c9da17f3929c\") " pod="openshift-dns/dns-default-b8fdl" Jul 09 14:15:02.191957 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.191936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9284783-b78b-4612-96b7-38fb17d6e4b4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-24tn7\" (UID: \"a9284783-b78b-4612-96b7-38fb17d6e4b4\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:15:02.289566 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.289534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:15:02.289707 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.289626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:15:02.292474 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.292447 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"image-registry-547b9bc97d-r5mw4\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:15:02.292607 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.292550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967aa6b7-9ffe-405d-93c6-fdbc4c9a3715-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-866jh\" (UID: \"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:15:02.309393 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.309368 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-57dt4\"" Jul 09 14:15:02.317368 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.317201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:15:02.343337 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.343315 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-twt95\"" Jul 09 14:15:02.351495 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.351463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" Jul 09 14:15:02.374597 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.374567 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w749s\"" Jul 09 14:15:02.381999 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.381970 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:02.383134 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.383100 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kggvr\"" Jul 09 14:15:02.390748 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.390715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8fdl" Jul 09 14:15:02.413992 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.413965 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v24rb\"" Jul 09 14:15:02.421327 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.421293 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" Jul 09 14:15:02.424655 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.424629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:15:02.479854 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.479819 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c47c5d5b5-pnpfj"] Jul 09 14:15:02.482255 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:02.482192 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d1a01d_bbcc_490c_a195_09158937207f.slice/crio-1106b9142fc5f0a350132f6a2ffb83d665e914c639f245c8c54b0fda256c3906 WatchSource:0}: Error finding container 1106b9142fc5f0a350132f6a2ffb83d665e914c639f245c8c54b0fda256c3906: Status 404 returned error can't find the container with id 1106b9142fc5f0a350132f6a2ffb83d665e914c639f245c8c54b0fda256c3906 Jul 09 14:15:02.515452 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.515353 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2gbj" event={"ID":"14e7daa9-1b76-445a-9b5a-b3106f89ff9d","Type":"ContainerStarted","Data":"010f365c145e33d6286d746f6a793d58c3972e04fa8da9b13a946dc5a7013888"} Jul 09 14:15:02.525776 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.525690 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" event={"ID":"e8d1a01d-bbcc-490c-a195-09158937207f","Type":"ContainerStarted","Data":"1106b9142fc5f0a350132f6a2ffb83d665e914c639f245c8c54b0fda256c3906"} Jul 09 14:15:02.552947 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.552868 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7"] Jul 09 14:15:02.569824 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:02.569694 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9284783_b78b_4612_96b7_38fb17d6e4b4.slice/crio-7fa04f10f4310eb88ca52348f94e06d76f6a0e91a55947e8f9bfa0524d7eccc4 WatchSource:0}: Error finding container 7fa04f10f4310eb88ca52348f94e06d76f6a0e91a55947e8f9bfa0524d7eccc4: Status 404 returned error can't find the container with id 7fa04f10f4310eb88ca52348f94e06d76f6a0e91a55947e8f9bfa0524d7eccc4 Jul 09 14:15:02.590754 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.590492 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74555698d-dkf2p"] Jul 09 14:15:02.595540 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:02.595506 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e281907_2c2d_4944_8736_22f20ce41bb7.slice/crio-364331216e046e56555756ad26f13be44992a1828de157161976515fa331ba51 WatchSource:0}: Error finding container 364331216e046e56555756ad26f13be44992a1828de157161976515fa331ba51: Status 404 returned error can't find the container with id 364331216e046e56555756ad26f13be44992a1828de157161976515fa331ba51 Jul 09 14:15:02.627899 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.627538 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8fdl"] Jul 09 14:15:02.633968 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:02.633919 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d25ba2_c533_44bd_be10_c9da17f3929c.slice/crio-1c9fc576aed19afee31e3fcd1f14913dc223b680901cc168dce8ebc8f5074316 WatchSource:0}: Error finding container 1c9fc576aed19afee31e3fcd1f14913dc223b680901cc168dce8ebc8f5074316: Status 404 returned error can't find the container with id 1c9fc576aed19afee31e3fcd1f14913dc223b680901cc168dce8ebc8f5074316 Jul 09 14:15:02.656199 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.656166 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-866jh"] Jul 09 14:15:02.660321 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:02.660270 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967aa6b7_9ffe_405d_93c6_fdbc4c9a3715.slice/crio-cd18d1749f63b4355ab40159fa16209b9a9c75372de5ae4c9f29cf104c115ec5 WatchSource:0}: Error finding container cd18d1749f63b4355ab40159fa16209b9a9c75372de5ae4c9f29cf104c115ec5: Status 404 returned error can't find the container with id cd18d1749f63b4355ab40159fa16209b9a9c75372de5ae4c9f29cf104c115ec5 Jul 09 14:15:02.694000 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:02.693953 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547b9bc97d-r5mw4"] Jul 09 14:15:02.699559 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:02.699521 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f86062_0ebd_4395_88d2_cca4bf6dd7dd.slice/crio-c8eaffb17b32b4e233df412e01a08f3e437fbe12a016369275b8bb876952994a WatchSource:0}: Error finding container c8eaffb17b32b4e233df412e01a08f3e437fbe12a016369275b8bb876952994a: Status 404 returned error can't find the container with id c8eaffb17b32b4e233df412e01a08f3e437fbe12a016369275b8bb876952994a Jul 09 14:15:03.531350 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.531299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" event={"ID":"e8d1a01d-bbcc-490c-a195-09158937207f","Type":"ContainerStarted","Data":"b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334"} Jul 09 14:15:03.531779 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.531450 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:15:03.532727 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.532700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" event={"ID":"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd","Type":"ContainerStarted","Data":"16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc"} Jul 09 14:15:03.532847 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.532745 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" event={"ID":"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd","Type":"ContainerStarted","Data":"c8eaffb17b32b4e233df412e01a08f3e437fbe12a016369275b8bb876952994a"} Jul 09 14:15:03.532847 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.532770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:15:03.533747 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.533727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" event={"ID":"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715","Type":"ContainerStarted","Data":"cd18d1749f63b4355ab40159fa16209b9a9c75372de5ae4c9f29cf104c115ec5"} Jul 09 14:15:03.534647 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.534627 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8fdl" event={"ID":"94d25ba2-c533-44bd-be10-c9da17f3929c","Type":"ContainerStarted","Data":"1c9fc576aed19afee31e3fcd1f14913dc223b680901cc168dce8ebc8f5074316"} Jul 09 14:15:03.536002 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.535983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74555698d-dkf2p" event={"ID":"7e281907-2c2d-4944-8736-22f20ce41bb7","Type":"ContainerStarted","Data":"25e81bc91968b78aebb79b4b3965625502ef36c9832f963d9e5f32dc4afbe33e"} Jul 09 14:15:03.536096 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.536015 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74555698d-dkf2p" event={"ID":"7e281907-2c2d-4944-8736-22f20ce41bb7","Type":"ContainerStarted","Data":"364331216e046e56555756ad26f13be44992a1828de157161976515fa331ba51"} Jul 09 14:15:03.537126 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.537106 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" event={"ID":"a9284783-b78b-4612-96b7-38fb17d6e4b4","Type":"ContainerStarted","Data":"7fa04f10f4310eb88ca52348f94e06d76f6a0e91a55947e8f9bfa0524d7eccc4"} Jul 09 14:15:03.575673 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.575625 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" podStartSLOduration=68.575606202 podStartE2EDuration="1m8.575606202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:03.555502323 +0000 UTC m=+70.244431562" watchObservedRunningTime="2026-07-09 14:15:03.575606202 +0000 UTC m=+70.264535440" Jul 09 14:15:03.575819 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.575777 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-74555698d-dkf2p" podStartSLOduration=42.575767447 podStartE2EDuration="42.575767447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:03.574548535 +0000 UTC m=+70.263477776" watchObservedRunningTime="2026-07-09 14:15:03.575767447 +0000 UTC m=+70.264696687" Jul 09 14:15:03.597911 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:03.597855 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" podStartSLOduration=36.597832408 podStartE2EDuration="36.597832408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:03.596034739 +0000 UTC m=+70.284964011" watchObservedRunningTime="2026-07-09 14:15:03.597832408 +0000 UTC m=+70.286761644" Jul 09 14:15:04.383003 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:04.382966 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:04.386698 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:04.386469 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:04.544304 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:04.544187 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jdld" event={"ID":"e810efc5-e567-4924-af70-18dd39817937","Type":"ContainerStarted","Data":"43f041f47a215ad6223da17d68f459c77243fec8965474478609dc0de53e02dc"} Jul 09 14:15:04.544304 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:04.544267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jdld" event={"ID":"e810efc5-e567-4924-af70-18dd39817937","Type":"ContainerStarted","Data":"5e805c23431e4d8e18bb926f58f512ff7b4675a399d1d8040a245d9ee6e9a221"} Jul 09 14:15:04.544829 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:04.544778 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:04.546205 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:04.546181 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74555698d-dkf2p" Jul 09 14:15:04.589126 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:04.589062 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8jdld" podStartSLOduration=67.470499758 podStartE2EDuration="1m9.58904149s" totalImagesPullingTime="2.118541732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:04.587546217 +0000 UTC m=+71.276475481" watchObservedRunningTime="2026-07-09 14:15:04.58904149 +0000 UTC m=+71.277970724" Jul 09 14:15:06.241470 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.241378 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2"] Jul 09 14:15:06.248169 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.248137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.251402 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.251373 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 14:15:06.251756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.251737 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 14:15:06.251845 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.251830 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 14:15:06.252027 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.252010 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 14:15:06.253003 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.252973 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 14:15:06.253237 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.253207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 14:15:06.253574 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.253465 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 14:15:06.266189 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.266152 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-547b9bc97d-r5mw4"] Jul 09 14:15:06.266375 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.266248 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2"] Jul 09 14:15:06.266375 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.266274 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r"] Jul 09 14:15:06.271313 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.271284 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899"] Jul 09 14:15:06.271652 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.271445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.280170 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.277358 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 09 14:15:06.281832 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.281812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.286555 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.286530 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-c49kw\"" Jul 09 14:15:06.286872 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.286852 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 09 14:15:06.294831 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.294796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899"] Jul 09 14:15:06.296966 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.296365 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r"] Jul 09 14:15:06.330689 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.330845 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee29f8c4-76cc-4ede-af54-e80a5aa12286-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.330845 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-hub\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.330845 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4d2f\" (UniqueName: \"kubernetes.io/projected/ee29f8c4-76cc-4ede-af54-e80a5aa12286-kube-api-access-s4d2f\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.330845 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwj9w\" (UniqueName: \"kubernetes.io/projected/56c2b500-4039-460f-87b1-89979858f349-kube-api-access-mwj9w\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.331054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c2b500-4039-460f-87b1-89979858f349-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.331054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee29f8c4-76cc-4ede-af54-e80a5aa12286-tmp\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.331054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19f769c0-f709-45d8-ad40-05641bc727c7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54998fd47b-cs899\" (UID: \"19f769c0-f709-45d8-ad40-05641bc727c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.331054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-ca\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.331054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.330994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.331054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.331026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klcf6\" (UniqueName: \"kubernetes.io/projected/19f769c0-f709-45d8-ad40-05641bc727c7-kube-api-access-klcf6\") pod \"managed-serviceaccount-addon-agent-54998fd47b-cs899\" (UID: \"19f769c0-f709-45d8-ad40-05641bc727c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.432170 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.432377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klcf6\" (UniqueName: \"kubernetes.io/projected/19f769c0-f709-45d8-ad40-05641bc727c7-kube-api-access-klcf6\") pod \"managed-serviceaccount-addon-agent-54998fd47b-cs899\" (UID: \"19f769c0-f709-45d8-ad40-05641bc727c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.432377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.432377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee29f8c4-76cc-4ede-af54-e80a5aa12286-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.432377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-hub\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.432377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4d2f\" (UniqueName: \"kubernetes.io/projected/ee29f8c4-76cc-4ede-af54-e80a5aa12286-kube-api-access-s4d2f\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.432631 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwj9w\" (UniqueName: \"kubernetes.io/projected/56c2b500-4039-460f-87b1-89979858f349-kube-api-access-mwj9w\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.432631 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c2b500-4039-460f-87b1-89979858f349-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.432631 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee29f8c4-76cc-4ede-af54-e80a5aa12286-tmp\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.432631 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19f769c0-f709-45d8-ad40-05641bc727c7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54998fd47b-cs899\" (UID: \"19f769c0-f709-45d8-ad40-05641bc727c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.432631 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.432527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-ca\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.435779 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.435152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee29f8c4-76cc-4ede-af54-e80a5aa12286-tmp\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.436385 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.436311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.436385 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.436325 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2"] Jul 09 14:15:06.437522 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.437492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c2b500-4039-460f-87b1-89979858f349-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.438136 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.438114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee29f8c4-76cc-4ede-af54-e80a5aa12286-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.439211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.439164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-hub\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.439693 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.439652 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.440737 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.440646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/56c2b500-4039-460f-87b1-89979858f349-ca\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.440838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.440743 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19f769c0-f709-45d8-ad40-05641bc727c7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54998fd47b-cs899\" (UID: \"19f769c0-f709-45d8-ad40-05641bc727c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.445677 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.444740 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:06.452437 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.450416 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kzvg7\"" Jul 09 14:15:06.452437 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.450693 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 09 14:15:06.452437 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.450921 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4d2f\" (UniqueName: \"kubernetes.io/projected/ee29f8c4-76cc-4ede-af54-e80a5aa12286-kube-api-access-s4d2f\") pod \"klusterlet-addon-workmgr-5f68b95fc5-b2x4r\" (UID: \"ee29f8c4-76cc-4ede-af54-e80a5aa12286\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.452437 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.451194 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2"] Jul 09 14:15:06.453147 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.453097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klcf6\" (UniqueName: \"kubernetes.io/projected/19f769c0-f709-45d8-ad40-05641bc727c7-kube-api-access-klcf6\") pod \"managed-serviceaccount-addon-agent-54998fd47b-cs899\" (UID: \"19f769c0-f709-45d8-ad40-05641bc727c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.459953 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.459921 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwj9w\" (UniqueName: \"kubernetes.io/projected/56c2b500-4039-460f-87b1-89979858f349-kube-api-access-mwj9w\") pod \"cluster-proxy-proxy-agent-6fcbd48f47-5xtz2\" (UID: \"56c2b500-4039-460f-87b1-89979858f349\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.533771 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.533705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4ae5f574-4d81-4450-a29d-4fb74f3626b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-jbhm2\" (UID: \"4ae5f574-4d81-4450-a29d-4fb74f3626b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:06.556923 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.556882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" event={"ID":"a9284783-b78b-4612-96b7-38fb17d6e4b4","Type":"ContainerStarted","Data":"5ea72140f5ec8eba9a1b57d56da6a6fe9e34bea742726327f8f831064d1252f1"} Jul 09 14:15:06.560213 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.560182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8fdl" event={"ID":"94d25ba2-c533-44bd-be10-c9da17f3929c","Type":"ContainerStarted","Data":"50f1ae0586bc265894f6f4627cc6b9da1c4440a6f80a208fdb50e085fccc6fbb"} Jul 09 14:15:06.560371 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.560286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8fdl" event={"ID":"94d25ba2-c533-44bd-be10-c9da17f3929c","Type":"ContainerStarted","Data":"f256ee79fe2e5051de8a3661093efa773b906f207adac53b07d21cc31cd91686"} Jul 09 14:15:06.560371 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.560322 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b8fdl" Jul 09 14:15:06.573436 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.573403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" Jul 09 14:15:06.581217 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.581072 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-24tn7" podStartSLOduration=42.48791564 podStartE2EDuration="45.581050637s" totalImagesPullingTime="3.093134997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:06.579357721 +0000 UTC m=+73.268286961" watchObservedRunningTime="2026-07-09 14:15:06.581050637 +0000 UTC m=+73.269979877" Jul 09 14:15:06.586004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.585950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:06.607930 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.607661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" Jul 09 14:15:06.636080 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:06.634899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4ae5f574-4d81-4450-a29d-4fb74f3626b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-jbhm2\" (UID: \"4ae5f574-4d81-4450-a29d-4fb74f3626b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:06.636279 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:15:06.636182 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 09 14:15:06.636279 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:15:06.636266 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae5f574-4d81-4450-a29d-4fb74f3626b9-tls-certificates podName:4ae5f574-4d81-4450-a29d-4fb74f3626b9 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:07.136245178 +0000 UTC m=+73.825174396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4ae5f574-4d81-4450-a29d-4fb74f3626b9-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-jbhm2" (UID: "4ae5f574-4d81-4450-a29d-4fb74f3626b9") : secret "prometheus-operator-admission-webhook-tls" not found Jul 09 14:15:07.139277 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:07.139213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4ae5f574-4d81-4450-a29d-4fb74f3626b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-jbhm2\" (UID: \"4ae5f574-4d81-4450-a29d-4fb74f3626b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:07.141952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:07.141921 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4ae5f574-4d81-4450-a29d-4fb74f3626b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-jbhm2\" (UID: \"4ae5f574-4d81-4450-a29d-4fb74f3626b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:07.393379 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:07.393291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:09.351589 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.351470 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b8fdl" podStartSLOduration=36.326940254 podStartE2EDuration="39.351453528s" totalImagesPullingTime="3.024513274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:06.608882593 +0000 UTC m=+73.297811833" watchObservedRunningTime="2026-07-09 14:15:09.351453528 +0000 UTC m=+76.040382783" Jul 09 14:15:09.352710 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.352666 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r"] Jul 09 14:15:09.356746 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:09.356715 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee29f8c4_76cc_4ede_af54_e80a5aa12286.slice/crio-5e4dc3f0fdf417dfc5a43ba8ea0f83af19cb34635583347cc28e9b7e0406148e WatchSource:0}: Error finding container 5e4dc3f0fdf417dfc5a43ba8ea0f83af19cb34635583347cc28e9b7e0406148e: Status 404 returned error can't find the container with id 5e4dc3f0fdf417dfc5a43ba8ea0f83af19cb34635583347cc28e9b7e0406148e Jul 09 14:15:09.364289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.364257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2"] Jul 09 14:15:09.368003 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:09.367936 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae5f574_4d81_4450_a29d_4fb74f3626b9.slice/crio-9441dc38c6bcfde43e74eca4427605ba7afff4994a5af2e163c698e113a8caea WatchSource:0}: Error finding container 9441dc38c6bcfde43e74eca4427605ba7afff4994a5af2e163c698e113a8caea: Status 404 returned error can't find the container with id 9441dc38c6bcfde43e74eca4427605ba7afff4994a5af2e163c698e113a8caea Jul 09 14:15:09.576661 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.576553 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2gbj" event={"ID":"14e7daa9-1b76-445a-9b5a-b3106f89ff9d","Type":"ContainerStarted","Data":"fff2366a9bf36c78f163f3a4c973d7a05dac9e45e1ddd7583d60b2d183d96970"} Jul 09 14:15:09.577826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.577799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" event={"ID":"ee29f8c4-76cc-4ede-af54-e80a5aa12286","Type":"ContainerStarted","Data":"5e4dc3f0fdf417dfc5a43ba8ea0f83af19cb34635583347cc28e9b7e0406148e"} Jul 09 14:15:09.579669 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.579619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" event={"ID":"967aa6b7-9ffe-405d-93c6-fdbc4c9a3715","Type":"ContainerStarted","Data":"1a2d05e6bf4f486de8d15ff04d145dea1b19fd199e35d6b5a7dd5dac780cbf2e"} Jul 09 14:15:09.580849 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.580820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" event={"ID":"4ae5f574-4d81-4450-a29d-4fb74f3626b9","Type":"ContainerStarted","Data":"9441dc38c6bcfde43e74eca4427605ba7afff4994a5af2e163c698e113a8caea"} Jul 09 14:15:09.582325 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.582200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bgmqg" event={"ID":"2cdcb585-98f7-4c44-928e-68b3dc41164e","Type":"ContainerStarted","Data":"0e309db815d487d9d2d1d71388cbe8a5aa9277130654bcf4644fb661cbd71cee"} Jul 09 14:15:09.600216 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.600158 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s2gbj" podStartSLOduration=17.307380697 podStartE2EDuration="28.600139644s" totalImagesPullingTime="11.292758947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:09.59863893 +0000 UTC m=+76.287568166" watchObservedRunningTime="2026-07-09 14:15:09.600139644 +0000 UTC m=+76.289068884" Jul 09 14:15:09.602368 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.602305 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899"] Jul 09 14:15:09.603547 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:09.603517 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19f769c0_f709_45d8_ad40_05641bc727c7.slice/crio-168330c029dc62a6b08eacb88343392cca5f044d0bfa442c614c346efaf49925 WatchSource:0}: Error finding container 168330c029dc62a6b08eacb88343392cca5f044d0bfa442c614c346efaf49925: Status 404 returned error can't find the container with id 168330c029dc62a6b08eacb88343392cca5f044d0bfa442c614c346efaf49925 Jul 09 14:15:09.603879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.603860 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2"] Jul 09 14:15:09.607633 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:09.607606 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56c2b500_4039_460f_87b1_89979858f349.slice/crio-d6a1b54a24b17c820ecc2e404cd55d645882f867d33a63734200ee7b874e5ecc WatchSource:0}: Error finding container d6a1b54a24b17c820ecc2e404cd55d645882f867d33a63734200ee7b874e5ecc: Status 404 returned error can't find the container with id d6a1b54a24b17c820ecc2e404cd55d645882f867d33a63734200ee7b874e5ecc Jul 09 14:15:09.615006 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.614953 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bgmqg" podStartSLOduration=66.875555694 podStartE2EDuration="1m14.614934612s" totalImagesPullingTime="7.739378918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:09.613822812 +0000 UTC m=+76.302752057" watchObservedRunningTime="2026-07-09 14:15:09.614934612 +0000 UTC m=+76.303863849" Jul 09 14:15:09.631540 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:09.631482 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-866jh" podStartSLOduration=34.135807298 podStartE2EDuration="40.63145562s" totalImagesPullingTime="6.495648322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:09.629486631 +0000 UTC m=+76.318415869" watchObservedRunningTime="2026-07-09 14:15:09.63145562 +0000 UTC m=+76.320384860" Jul 09 14:15:10.590969 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:10.590878 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" event={"ID":"19f769c0-f709-45d8-ad40-05641bc727c7","Type":"ContainerStarted","Data":"168330c029dc62a6b08eacb88343392cca5f044d0bfa442c614c346efaf49925"} Jul 09 14:15:10.593909 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:10.593875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" event={"ID":"56c2b500-4039-460f-87b1-89979858f349","Type":"ContainerStarted","Data":"d6a1b54a24b17c820ecc2e404cd55d645882f867d33a63734200ee7b874e5ecc"} Jul 09 14:15:11.880389 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:11.880349 2575 scope.go:117] "RemoveContainer" containerID="765ed366daee50c5c17c44b321866fca19522fdd85f13585a9ebc1a43941967e" Jul 09 14:15:13.611759 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.611705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" event={"ID":"4ae5f574-4d81-4450-a29d-4fb74f3626b9","Type":"ContainerStarted","Data":"82003d83e588ce78355a9144c40f6f266168cc4d64b7a8656c5b46adf048aedb"} Jul 09 14:15:13.612300 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.611773 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:13.613994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.613974 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:15:13.614100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.614031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wxw4j" event={"ID":"66bca9b3-d1e5-475b-80a2-824250d8ae0f","Type":"ContainerStarted","Data":"7239989f9e0559cfc9802c5dd3ce37ea326b5368cadbf5a002044eebb454e3a4"} Jul 09 14:15:13.614492 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.614470 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:15:13.617336 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.617295 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" Jul 09 14:15:13.620679 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.620540 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-wxw4j" Jul 09 14:15:13.630302 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.629359 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-jbhm2" podStartSLOduration=4.341361336 podStartE2EDuration="7.629345102s" totalImagesPullingTime="3.287983766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:13.627828065 +0000 UTC m=+80.316757304" watchObservedRunningTime="2026-07-09 14:15:13.629345102 +0000 UTC m=+80.318274319" Jul 09 14:15:13.648093 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.648028 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-wxw4j" podStartSLOduration=43.786221258 podStartE2EDuration="52.64801386s" totalImagesPullingTime="8.861792602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:13.646181054 +0000 UTC m=+80.335110309" watchObservedRunningTime="2026-07-09 14:15:13.64801386 +0000 UTC m=+80.336943098" Jul 09 14:15:13.783442 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.783402 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-mtpth"] Jul 09 14:15:13.806943 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.806903 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-mtpth"] Jul 09 14:15:13.807133 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.807045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-mtpth" Jul 09 14:15:13.811153 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.810020 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 14:15:13.811153 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.810093 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wcp44\"" Jul 09 14:15:13.811153 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.810304 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 14:15:13.901585 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:13.901495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk5v\" (UniqueName: \"kubernetes.io/projected/ec42a2b9-ace3-48cd-b822-411e206e717d-kube-api-access-pfk5v\") pod \"downloads-68c9cbb497-mtpth\" (UID: \"ec42a2b9-ace3-48cd-b822-411e206e717d\") " pod="openshift-console/downloads-68c9cbb497-mtpth" Jul 09 14:15:14.001995 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.001959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk5v\" (UniqueName: \"kubernetes.io/projected/ec42a2b9-ace3-48cd-b822-411e206e717d-kube-api-access-pfk5v\") pod \"downloads-68c9cbb497-mtpth\" (UID: \"ec42a2b9-ace3-48cd-b822-411e206e717d\") " pod="openshift-console/downloads-68c9cbb497-mtpth" Jul 09 14:15:14.011240 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.011193 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk5v\" (UniqueName: \"kubernetes.io/projected/ec42a2b9-ace3-48cd-b822-411e206e717d-kube-api-access-pfk5v\") pod \"downloads-68c9cbb497-mtpth\" (UID: \"ec42a2b9-ace3-48cd-b822-411e206e717d\") " pod="openshift-console/downloads-68c9cbb497-mtpth" Jul 09 14:15:14.120338 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.120300 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-mtpth" Jul 09 14:15:14.528404 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.528364 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ls2lj"] Jul 09 14:15:14.545031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.544998 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ls2lj"] Jul 09 14:15:14.545211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.545122 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.549926 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.549904 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 14:15:14.549926 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.549928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 14:15:14.550124 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.550060 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 14:15:14.550312 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.550196 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r7z52\"" Jul 09 14:15:14.608333 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.608290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.608523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.608341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.608523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.608428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxznr\" (UniqueName: \"kubernetes.io/projected/101345db-9986-43e2-acf5-856f6e0f6610-kube-api-access-qxznr\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.608652 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.608536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/101345db-9986-43e2-acf5-856f6e0f6610-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.710014 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.709964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/101345db-9986-43e2-acf5-856f6e0f6610-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.710526 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.710475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.710526 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.710509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.710710 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.710548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxznr\" (UniqueName: \"kubernetes.io/projected/101345db-9986-43e2-acf5-856f6e0f6610-kube-api-access-qxznr\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.711181 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.711152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/101345db-9986-43e2-acf5-856f6e0f6610-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.711181 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:15:14.711186 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 09 14:15:14.711397 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:15:14.711275 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-tls podName:101345db-9986-43e2-acf5-856f6e0f6610 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:15.211253645 +0000 UTC m=+81.900182868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-tls") pod "prometheus-operator-64785b6675-ls2lj" (UID: "101345db-9986-43e2-acf5-856f6e0f6610") : secret "prometheus-operator-tls" not found Jul 09 14:15:14.713882 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.713859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:14.720742 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:14.720716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxznr\" (UniqueName: \"kubernetes.io/projected/101345db-9986-43e2-acf5-856f6e0f6610-kube-api-access-qxznr\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:15.216248 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:15.216193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:15.232381 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:15.232340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/101345db-9986-43e2-acf5-856f6e0f6610-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ls2lj\" (UID: \"101345db-9986-43e2-acf5-856f6e0f6610\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:15.458583 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:15.458541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" Jul 09 14:15:15.986935 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:15.986856 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ls2lj"] Jul 09 14:15:15.990593 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:15.990565 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101345db_9986_43e2_acf5_856f6e0f6610.slice/crio-c1a45dd249ca9e7ca35a4f7b148b82d36bf8983452604fb2b4add090e0f384c1 WatchSource:0}: Error finding container c1a45dd249ca9e7ca35a4f7b148b82d36bf8983452604fb2b4add090e0f384c1: Status 404 returned error can't find the container with id c1a45dd249ca9e7ca35a4f7b148b82d36bf8983452604fb2b4add090e0f384c1 Jul 09 14:15:16.008307 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.008243 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-mtpth"] Jul 09 14:15:16.012133 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:16.012101 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec42a2b9_ace3_48cd_b822_411e206e717d.slice/crio-f3bc95a0dea318e759357f483357941689ba0b49692eba5f2441ff437f65ea53 WatchSource:0}: Error finding container f3bc95a0dea318e759357f483357941689ba0b49692eba5f2441ff437f65ea53: Status 404 returned error can't find the container with id f3bc95a0dea318e759357f483357941689ba0b49692eba5f2441ff437f65ea53 Jul 09 14:15:16.566853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.566821 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b8fdl" Jul 09 14:15:16.630128 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.630085 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" event={"ID":"56c2b500-4039-460f-87b1-89979858f349","Type":"ContainerStarted","Data":"0ec248dcae23627783a8f1f0267989ca6f8cd534ba816d5a38895fe266515a11"} Jul 09 14:15:16.630329 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.630137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" event={"ID":"56c2b500-4039-460f-87b1-89979858f349","Type":"ContainerStarted","Data":"61388dcc0947f06e0684ff89dd1e5387b7a47b1dea8653024b6879fe0f5cb2d4"} Jul 09 14:15:16.630329 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.630152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" event={"ID":"56c2b500-4039-460f-87b1-89979858f349","Type":"ContainerStarted","Data":"cb50d565e1b2e98fe26fcbc6738d787bb0b8122d9ca9bf5fd06a8eec8e123a02"} Jul 09 14:15:16.632284 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.632257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" event={"ID":"101345db-9986-43e2-acf5-856f6e0f6610","Type":"ContainerStarted","Data":"c1a45dd249ca9e7ca35a4f7b148b82d36bf8983452604fb2b4add090e0f384c1"} Jul 09 14:15:16.633932 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.633896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-mtpth" event={"ID":"ec42a2b9-ace3-48cd-b822-411e206e717d","Type":"ContainerStarted","Data":"f3bc95a0dea318e759357f483357941689ba0b49692eba5f2441ff437f65ea53"} Jul 09 14:15:16.636397 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.636375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" event={"ID":"ee29f8c4-76cc-4ede-af54-e80a5aa12286","Type":"ContainerStarted","Data":"ab64a9314cdabcfb9fcfe093be121c48e612e92012ed6eb3a018ad0d23dbdc3a"} Jul 09 14:15:16.636588 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.636559 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:16.638372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.638352 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" Jul 09 14:15:16.648566 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.648516 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbd48f47-5xtz2" podStartSLOduration=4.4068757210000005 podStartE2EDuration="10.648499797s" totalImagesPullingTime="6.241624076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:16.64765581 +0000 UTC m=+83.336585050" watchObservedRunningTime="2026-07-09 14:15:16.648499797 +0000 UTC m=+83.337429033" Jul 09 14:15:16.665399 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:16.665324 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f68b95fc5-b2x4r" podStartSLOduration=4.18696609 podStartE2EDuration="10.665302032s" totalImagesPullingTime="6.478335942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:16.663576218 +0000 UTC m=+83.352505452" watchObservedRunningTime="2026-07-09 14:15:16.665302032 +0000 UTC m=+83.354231272" Jul 09 14:15:17.642423 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:17.642393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" event={"ID":"101345db-9986-43e2-acf5-856f6e0f6610","Type":"ContainerStarted","Data":"1921b9a26dd3e4ffbc6650d14d4c1b347ba2f1634277c849c2ee138e2f916741"} Jul 09 14:15:18.650127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:18.650088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" event={"ID":"101345db-9986-43e2-acf5-856f6e0f6610","Type":"ContainerStarted","Data":"ac9cb06114d2600721c4eb3efa65b20c63fbc6e3a18ef6b733fddb47a1a5e355"} Jul 09 14:15:18.669085 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:18.669031 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-ls2lj" podStartSLOduration=3.129878947 podStartE2EDuration="4.669014358s" totalImagesPullingTime="1.539135411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:18.668767336 +0000 UTC m=+85.357696577" watchObservedRunningTime="2026-07-09 14:15:18.669014358 +0000 UTC m=+85.357943574" Jul 09 14:15:20.970283 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:20.970250 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5k8cm"] Jul 09 14:15:20.975205 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:20.975178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:20.977636 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:20.977612 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 14:15:20.978067 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:20.978047 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fjdmz\"" Jul 09 14:15:20.978161 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:20.978049 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 14:15:20.978161 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:20.978104 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 14:15:21.068643 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-tls\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.068837 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-textfile\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.068837 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068721 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shnv\" (UniqueName: \"kubernetes.io/projected/818beee3-d1d7-495b-82db-bd1c0559a006-kube-api-access-6shnv\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.068837 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-root\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.068837 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.068837 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-accelerators-collector-config\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.068837 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-sys\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.069143 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-wtmp\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.069143 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.068924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/818beee3-d1d7-495b-82db-bd1c0559a006-metrics-client-ca\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-accelerators-collector-config\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-sys\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-wtmp\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/818beee3-d1d7-495b-82db-bd1c0559a006-metrics-client-ca\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-tls\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-sys\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.170591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-wtmp\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.171129 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-textfile\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.171129 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shnv\" (UniqueName: \"kubernetes.io/projected/818beee3-d1d7-495b-82db-bd1c0559a006-kube-api-access-6shnv\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.171129 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.170953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-root\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.171129 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.171050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/818beee3-d1d7-495b-82db-bd1c0559a006-root\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.171352 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.171337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-textfile\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.171412 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.171397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/818beee3-d1d7-495b-82db-bd1c0559a006-metrics-client-ca\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.171884 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.171855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-accelerators-collector-config\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.173692 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.173665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.176374 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.176350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/818beee3-d1d7-495b-82db-bd1c0559a006-node-exporter-tls\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.183077 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.183050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shnv\" (UniqueName: \"kubernetes.io/projected/818beee3-d1d7-495b-82db-bd1c0559a006-kube-api-access-6shnv\") pod \"node-exporter-5k8cm\" (UID: \"818beee3-d1d7-495b-82db-bd1c0559a006\") " pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.287664 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.287626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5k8cm" Jul 09 14:15:21.303451 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:15:21.303412 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818beee3_d1d7_495b_82db_bd1c0559a006.slice/crio-f68e966220d5f8218ab7738a6f2a9ab1bb87ae0f51c5b264bd7751f32dc051d3 WatchSource:0}: Error finding container f68e966220d5f8218ab7738a6f2a9ab1bb87ae0f51c5b264bd7751f32dc051d3: Status 404 returned error can't find the container with id f68e966220d5f8218ab7738a6f2a9ab1bb87ae0f51c5b264bd7751f32dc051d3 Jul 09 14:15:21.670966 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:21.670879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5k8cm" event={"ID":"818beee3-d1d7-495b-82db-bd1c0559a006","Type":"ContainerStarted","Data":"f68e966220d5f8218ab7738a6f2a9ab1bb87ae0f51c5b264bd7751f32dc051d3"} Jul 09 14:15:22.323346 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:22.323306 2575 patch_prober.go:28] interesting pod/image-registry-c47c5d5b5-pnpfj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 14:15:22.323822 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:22.323370 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" podUID="e8d1a01d-bbcc-490c-a195-09158937207f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 14:15:24.549969 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:24.549941 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:15:24.689679 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:24.689641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5k8cm" event={"ID":"818beee3-d1d7-495b-82db-bd1c0559a006","Type":"ContainerStarted","Data":"89f663b7d817152492a229211eb3b5910642f0221247a1fa90df8564a285d8ff"} Jul 09 14:15:25.696465 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:25.696367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" event={"ID":"19f769c0-f709-45d8-ad40-05641bc727c7","Type":"ContainerStarted","Data":"1718ff9bb4f34f9cd85ac76021457c9d784531fa4f9c826b5602c3d29ab684ce"} Jul 09 14:15:25.698600 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:25.698570 2575 generic.go:356] "Generic (PLEG): container finished" podID="818beee3-d1d7-495b-82db-bd1c0559a006" containerID="89f663b7d817152492a229211eb3b5910642f0221247a1fa90df8564a285d8ff" exitCode=0 Jul 09 14:15:25.698728 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:25.698621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5k8cm" event={"ID":"818beee3-d1d7-495b-82db-bd1c0559a006","Type":"ContainerDied","Data":"89f663b7d817152492a229211eb3b5910642f0221247a1fa90df8564a285d8ff"} Jul 09 14:15:25.720081 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:25.720014 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54998fd47b-cs899" podStartSLOduration=4.046981042 podStartE2EDuration="19.719993707s" totalImagesPullingTime="15.673012665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:25.718033847 +0000 UTC m=+92.406963088" watchObservedRunningTime="2026-07-09 14:15:25.719993707 +0000 UTC m=+92.408922957" Jul 09 14:15:26.274033 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:26.273998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:15:26.705280 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:26.705170 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5k8cm" event={"ID":"818beee3-d1d7-495b-82db-bd1c0559a006","Type":"ContainerStarted","Data":"c02b92c3d15773fd085ee5dd0e5823673113c2922b3fe5eac4b2a4ee2386e73f"} Jul 09 14:15:26.705280 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:26.705256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5k8cm" event={"ID":"818beee3-d1d7-495b-82db-bd1c0559a006","Type":"ContainerStarted","Data":"bb4f6d53a83a9d74dc31c3f26f7b6b00fef4d6a5e1c67fa29e0a35574666df8e"} Jul 09 14:15:26.729780 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:26.729723 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5k8cm" podStartSLOduration=3.3600554320000002 podStartE2EDuration="6.729707989s" totalImagesPullingTime="3.285051655s" totalInitContainerRuntime="84.600902ms" isStatefulPod=true podCreationTimestamp="2026-07-09 14:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:26.727568481 +0000 UTC m=+93.416497745" watchObservedRunningTime="2026-07-09 14:15:26.729707989 +0000 UTC m=+93.418637228" Jul 09 14:15:31.315613 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:31.315539 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" podUID="c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" containerName="registry" containerID="cri-o://16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc" gracePeriod=30 Jul 09 14:15:32.532811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:32.532780 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lfhcd" Jul 09 14:15:36.268115 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:36.268073 2575 patch_prober.go:28] interesting pod/image-registry-547b9bc97d-r5mw4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.15:5000/healthz\": dial tcp 10.132.0.15:5000: connect: connection refused" start-of-body= Jul 09 14:15:36.268608 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:36.268166 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" podUID="c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.15:5000/healthz\": dial tcp 10.132.0.15:5000: connect: connection refused" Jul 09 14:15:37.097744 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.097715 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:15:37.225131 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225043 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-certificates\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225131 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225107 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-bound-sa-token\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225138 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8hbw\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-kube-api-access-d8hbw\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225176 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-ca-trust-extracted\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225212 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-trusted-ca\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225250 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-installation-pull-secrets\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-image-registry-private-configuration\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225372 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225357 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") pod \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\" (UID: \"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd\") " Jul 09 14:15:37.225685 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.225654 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:15:37.226318 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.226206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:15:37.228398 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.228364 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:15:37.228578 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.228433 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:15:37.228578 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.228553 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:15:37.228728 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.228624 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-kube-api-access-d8hbw" (OuterVolumeSpecName: "kube-api-access-d8hbw") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "kube-api-access-d8hbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:15:37.229051 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.229026 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:15:37.230652 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.230630 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c47c5d5b5-pnpfj"] Jul 09 14:15:37.235313 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.235271 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" (UID: "c4f86062-0ebd-4395-88d2-cca4bf6dd7dd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:15:37.326734 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326688 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.327212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326750 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-registry-certificates\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.327212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326773 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-bound-sa-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.327212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326783 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8hbw\" (UniqueName: \"kubernetes.io/projected/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-kube-api-access-d8hbw\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.327212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326791 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-ca-trust-extracted\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.327212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326800 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-trusted-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.327212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326808 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-installation-pull-secrets\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.327212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.326817 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd-image-registry-private-configuration\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:15:37.768203 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.768166 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-mtpth" event={"ID":"ec42a2b9-ace3-48cd-b822-411e206e717d","Type":"ContainerStarted","Data":"732d444235ef690b178c2b69b90487f0ccf14b6ac3e96584e7813e0208c85f2a"} Jul 09 14:15:37.768417 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.768288 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-mtpth" Jul 09 14:15:37.769573 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.769545 2575 generic.go:356] "Generic (PLEG): container finished" podID="c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" containerID="16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc" exitCode=0 Jul 09 14:15:37.769702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.769597 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" event={"ID":"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd","Type":"ContainerDied","Data":"16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc"} Jul 09 14:15:37.769702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.769633 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" event={"ID":"c4f86062-0ebd-4395-88d2-cca4bf6dd7dd","Type":"ContainerDied","Data":"c8eaffb17b32b4e233df412e01a08f3e437fbe12a016369275b8bb876952994a"} Jul 09 14:15:37.769702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.769659 2575 scope.go:117] "RemoveContainer" containerID="16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc" Jul 09 14:15:37.769702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.769601 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547b9bc97d-r5mw4" Jul 09 14:15:37.769997 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.769970 2575 patch_prober.go:28] interesting pod/downloads-68c9cbb497-mtpth container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.27:8080/\": dial tcp 10.132.0.27:8080: connect: connection refused" start-of-body= Jul 09 14:15:37.770112 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.770020 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-mtpth" podUID="ec42a2b9-ace3-48cd-b822-411e206e717d" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.27:8080/\": dial tcp 10.132.0.27:8080: connect: connection refused" Jul 09 14:15:37.778601 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.778586 2575 scope.go:117] "RemoveContainer" containerID="16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc" Jul 09 14:15:37.778888 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:15:37.778869 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc\": container with ID starting with 16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc not found: ID does not exist" containerID="16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc" Jul 09 14:15:37.778935 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.778897 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc"} err="failed to get container status \"16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc\": rpc error: code = NotFound desc = could not find container \"16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc\": container with ID starting with 16cddfd7cbaf2c72f666bf06bd2d49d1d1160f650d7cdaf2a45beb42704363cc not found: ID does not exist" Jul 09 14:15:37.790102 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.790058 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-mtpth" podStartSLOduration=3.463047304 podStartE2EDuration="24.790044716s" totalImagesPullingTime="21.326997412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:37.789466503 +0000 UTC m=+104.478395742" watchObservedRunningTime="2026-07-09 14:15:37.790044716 +0000 UTC m=+104.478974019" Jul 09 14:15:37.805907 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.805873 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-547b9bc97d-r5mw4"] Jul 09 14:15:37.810283 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.810256 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-547b9bc97d-r5mw4"] Jul 09 14:15:37.887247 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:37.887189 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" path="/var/lib/kubelet/pods/c4f86062-0ebd-4395-88d2-cca4bf6dd7dd/volumes" Jul 09 14:15:38.791907 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:38.791863 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-mtpth" Jul 09 14:15:45.814879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:45.814848 2575 generic.go:356] "Generic (PLEG): container finished" podID="3506c053-ea2a-4484-b5c3-369c03d8baed" containerID="9c9b2be2511a67cb2d3fa67c7161d98fa4ae9cc5332ccf9b311c7573744e88b5" exitCode=0 Jul 09 14:15:45.815376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:45.814952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" event={"ID":"3506c053-ea2a-4484-b5c3-369c03d8baed","Type":"ContainerDied","Data":"9c9b2be2511a67cb2d3fa67c7161d98fa4ae9cc5332ccf9b311c7573744e88b5"} Jul 09 14:15:45.815376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:45.815363 2575 scope.go:117] "RemoveContainer" containerID="9c9b2be2511a67cb2d3fa67c7161d98fa4ae9cc5332ccf9b311c7573744e88b5" Jul 09 14:15:46.821201 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:15:46.821165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qnm92" event={"ID":"3506c053-ea2a-4484-b5c3-369c03d8baed","Type":"ContainerStarted","Data":"f9c67889e3c0740628a2db76e687d3a0be890c1b6e75fc7dff192a7feefa2796"} Jul 09 14:16:02.258239 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.258181 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" podUID="e8d1a01d-bbcc-490c-a195-09158937207f" containerName="registry" containerID="cri-o://b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334" gracePeriod=30 Jul 09 14:16:02.554737 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.554704 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:16:02.648579 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648539 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.648579 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648583 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-bound-sa-token\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.648812 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648610 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-registry-certificates\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.648812 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648632 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-trusted-ca\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.648812 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648658 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d1a01d-bbcc-490c-a195-09158937207f-ca-trust-extracted\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.648812 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648686 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-installation-pull-secrets\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.648812 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648746 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-image-registry-private-configuration\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.648812 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.648792 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh4nj\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-kube-api-access-gh4nj\") pod \"e8d1a01d-bbcc-490c-a195-09158937207f\" (UID: \"e8d1a01d-bbcc-490c-a195-09158937207f\") " Jul 09 14:16:02.649278 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.649150 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:02.649396 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.649335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:02.651080 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.651049 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:16:02.651254 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.651196 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:02.651411 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.651377 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:16:02.651558 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.651528 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:02.651621 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.651598 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-kube-api-access-gh4nj" (OuterVolumeSpecName: "kube-api-access-gh4nj") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "kube-api-access-gh4nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:16:02.657360 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.657327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d1a01d-bbcc-490c-a195-09158937207f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e8d1a01d-bbcc-490c-a195-09158937207f" (UID: "e8d1a01d-bbcc-490c-a195-09158937207f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:16:02.750271 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750211 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-registry-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.750271 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750267 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-bound-sa-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.750271 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750275 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-registry-certificates\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.750271 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750285 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d1a01d-bbcc-490c-a195-09158937207f-trusted-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.750525 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750294 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d1a01d-bbcc-490c-a195-09158937207f-ca-trust-extracted\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.750525 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750302 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-installation-pull-secrets\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.750525 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750313 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d1a01d-bbcc-490c-a195-09158937207f-image-registry-private-configuration\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.750525 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.750322 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh4nj\" (UniqueName: \"kubernetes.io/projected/e8d1a01d-bbcc-490c-a195-09158937207f-kube-api-access-gh4nj\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:16:02.895789 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.895756 2575 generic.go:356] "Generic (PLEG): container finished" podID="e8d1a01d-bbcc-490c-a195-09158937207f" containerID="b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334" exitCode=0 Jul 09 14:16:02.895952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.895812 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" Jul 09 14:16:02.895952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.895841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" event={"ID":"e8d1a01d-bbcc-490c-a195-09158937207f","Type":"ContainerDied","Data":"b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334"} Jul 09 14:16:02.895952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.895922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c47c5d5b5-pnpfj" event={"ID":"e8d1a01d-bbcc-490c-a195-09158937207f","Type":"ContainerDied","Data":"1106b9142fc5f0a350132f6a2ffb83d665e914c639f245c8c54b0fda256c3906"} Jul 09 14:16:02.895952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.895947 2575 scope.go:117] "RemoveContainer" containerID="b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334" Jul 09 14:16:02.905089 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.905069 2575 scope.go:117] "RemoveContainer" containerID="b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334" Jul 09 14:16:02.905419 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:16:02.905400 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334\": container with ID starting with b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334 not found: ID does not exist" containerID="b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334" Jul 09 14:16:02.905488 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.905427 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334"} err="failed to get container status \"b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334\": rpc error: code = NotFound desc = could not find container \"b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334\": container with ID starting with b3d33abde1c94fa093c848124bbc454ad17bdf648499532691b049ab3d017334 not found: ID does not exist" Jul 09 14:16:02.917682 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.917653 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c47c5d5b5-pnpfj"] Jul 09 14:16:02.927830 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:02.927799 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c47c5d5b5-pnpfj"] Jul 09 14:16:03.885466 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:03.885432 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d1a01d-bbcc-490c-a195-09158937207f" path="/var/lib/kubelet/pods/e8d1a01d-bbcc-490c-a195-09158937207f/volumes" Jul 09 14:16:05.909624 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:05.909591 2575 generic.go:356] "Generic (PLEG): container finished" podID="7d6410bb-ed3b-480e-bd9f-1bdef3e5e743" containerID="04990c7141f0e630954e679f62c8a636f9f22020700221b5c3fb06ae181f4989" exitCode=0 Jul 09 14:16:05.910093 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:05.909669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" event={"ID":"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743","Type":"ContainerDied","Data":"04990c7141f0e630954e679f62c8a636f9f22020700221b5c3fb06ae181f4989"} Jul 09 14:16:05.910161 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:05.910101 2575 scope.go:117] "RemoveContainer" containerID="04990c7141f0e630954e679f62c8a636f9f22020700221b5c3fb06ae181f4989" Jul 09 14:16:06.915861 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:06.915823 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-8jz5b" event={"ID":"7d6410bb-ed3b-480e-bd9f-1bdef3e5e743","Type":"ContainerStarted","Data":"d9f1a9c6cb323d6ca0844a158c6d57c1d5edde45015f5df59bdc75204c26c2dd"} Jul 09 14:16:15.955348 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:15.955313 2575 generic.go:356] "Generic (PLEG): container finished" podID="5a3562bb-92a7-4cc1-91bf-9df604d84f92" containerID="cb232800c7088617e361280e888a426d754c2f04d108a834f58372956d355265" exitCode=0 Jul 09 14:16:15.955765 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:15.955375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" event={"ID":"5a3562bb-92a7-4cc1-91bf-9df604d84f92","Type":"ContainerDied","Data":"cb232800c7088617e361280e888a426d754c2f04d108a834f58372956d355265"} Jul 09 14:16:15.955765 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:15.955687 2575 scope.go:117] "RemoveContainer" containerID="cb232800c7088617e361280e888a426d754c2f04d108a834f58372956d355265" Jul 09 14:16:16.961639 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:16:16.961604 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qptsb" event={"ID":"5a3562bb-92a7-4cc1-91bf-9df604d84f92","Type":"ContainerStarted","Data":"fd2894addef401ee2fe555369f77ae7843033ffd0f7d818d14f081518ea6d138"} Jul 09 14:18:16.467072 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.466993 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv"] Jul 09 14:18:16.467548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.467345 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d1a01d-bbcc-490c-a195-09158937207f" containerName="registry" Jul 09 14:18:16.467548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.467357 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d1a01d-bbcc-490c-a195-09158937207f" containerName="registry" Jul 09 14:18:16.467548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.467374 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" containerName="registry" Jul 09 14:18:16.467548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.467380 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" containerName="registry" Jul 09 14:18:16.467548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.467438 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f86062-0ebd-4395-88d2-cca4bf6dd7dd" containerName="registry" Jul 09 14:18:16.467548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.467447 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d1a01d-bbcc-490c-a195-09158937207f" containerName="registry" Jul 09 14:18:16.470470 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.470450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.473168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.473149 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 14:18:16.473280 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.473168 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 14:18:16.473514 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.473499 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 14:18:16.473560 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.473537 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 14:18:16.473600 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.473572 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qf47d\"" Jul 09 14:18:16.488431 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.488378 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv"] Jul 09 14:18:16.570998 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.570965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19b30445-5808-421c-a24c-715d93a47cb8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.571163 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.571013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19b30445-5808-421c-a24c-715d93a47cb8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.571163 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.571043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19b30445-5808-421c-a24c-715d93a47cb8-manifests\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.571163 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.571089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19b30445-5808-421c-a24c-715d93a47cb8-tmp\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.571163 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.571135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9k2d\" (UniqueName: \"kubernetes.io/projected/19b30445-5808-421c-a24c-715d93a47cb8-kube-api-access-q9k2d\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.672564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.672530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19b30445-5808-421c-a24c-715d93a47cb8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.672564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.672567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19b30445-5808-421c-a24c-715d93a47cb8-manifests\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.672775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.672590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19b30445-5808-421c-a24c-715d93a47cb8-tmp\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.672775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.672618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9k2d\" (UniqueName: \"kubernetes.io/projected/19b30445-5808-421c-a24c-715d93a47cb8-kube-api-access-q9k2d\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.672775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.672666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19b30445-5808-421c-a24c-715d93a47cb8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.672999 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.672976 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19b30445-5808-421c-a24c-715d93a47cb8-manifests\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.674999 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.674968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19b30445-5808-421c-a24c-715d93a47cb8-tmp\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.675194 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.675177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19b30445-5808-421c-a24c-715d93a47cb8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.675261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.675247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19b30445-5808-421c-a24c-715d93a47cb8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.681033 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.681011 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9k2d\" (UniqueName: \"kubernetes.io/projected/19b30445-5808-421c-a24c-715d93a47cb8-kube-api-access-q9k2d\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-6f4fv\" (UID: \"19b30445-5808-421c-a24c-715d93a47cb8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.781152 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.781115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:16.918159 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:16.918134 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv"] Jul 09 14:18:16.920738 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:18:16.920710 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b30445_5808_421c_a24c_715d93a47cb8.slice/crio-d5db77cfefb1e3973180e32400c577659d39d6cf44fa81ff189839cad5d38efb WatchSource:0}: Error finding container d5db77cfefb1e3973180e32400c577659d39d6cf44fa81ff189839cad5d38efb: Status 404 returned error can't find the container with id d5db77cfefb1e3973180e32400c577659d39d6cf44fa81ff189839cad5d38efb Jul 09 14:18:17.439977 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:17.439937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" event={"ID":"19b30445-5808-421c-a24c-715d93a47cb8","Type":"ContainerStarted","Data":"d5db77cfefb1e3973180e32400c577659d39d6cf44fa81ff189839cad5d38efb"} Jul 09 14:18:20.455817 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:20.455783 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" event={"ID":"19b30445-5808-421c-a24c-715d93a47cb8","Type":"ContainerStarted","Data":"d643cb77c99313667373ab149ab50680c7c5db9cdbb6c9f7c2c0059f25f21436"} Jul 09 14:18:21.461766 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:21.461677 2575 generic.go:356] "Generic (PLEG): container finished" podID="19b30445-5808-421c-a24c-715d93a47cb8" containerID="d643cb77c99313667373ab149ab50680c7c5db9cdbb6c9f7c2c0059f25f21436" exitCode=0 Jul 09 14:18:21.461766 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:21.461737 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" event={"ID":"19b30445-5808-421c-a24c-715d93a47cb8","Type":"ContainerDied","Data":"d643cb77c99313667373ab149ab50680c7c5db9cdbb6c9f7c2c0059f25f21436"} Jul 09 14:18:22.467619 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:22.467577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" event={"ID":"19b30445-5808-421c-a24c-715d93a47cb8","Type":"ContainerStarted","Data":"6cbb294f627444273d40d979dbc2048ef63207631c0175204be009dc6be895b2"} Jul 09 14:18:22.468117 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:22.467666 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:22.495860 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:22.495420 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" podStartSLOduration=2.050424253 podStartE2EDuration="6.495388161s" totalImagesPullingTime="3.366150819s" totalInitContainerRuntime="1.078813089s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:22.493655667 +0000 UTC m=+269.182584931" watchObservedRunningTime="2026-07-09 14:18:22.495388161 +0000 UTC m=+269.184317403" Jul 09 14:18:33.475044 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:33.475012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-6f4fv" Jul 09 14:18:36.679101 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.679064 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm"] Jul 09 14:18:36.692394 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.692369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.697110 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.697085 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-slmz9\"" Jul 09 14:18:36.697256 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.697088 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 14:18:36.698242 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.698199 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 14:18:36.706161 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.706139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm"] Jul 09 14:18:36.846530 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.846495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d09296b9-b36a-4945-88be-6f70490e6e8c-tmp\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.846782 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.846556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w969c\" (UniqueName: \"kubernetes.io/projected/d09296b9-b36a-4945-88be-6f70490e6e8c-kube-api-access-w969c\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.846782 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.846654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09296b9-b36a-4945-88be-6f70490e6e8c-tls-certs\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.881162 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.881125 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fk49v"] Jul 09 14:18:36.884514 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.884497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:36.898471 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.898440 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-btf4h\"" Jul 09 14:18:36.898872 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.898846 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 14:18:36.920168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.920135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fk49v"] Jul 09 14:18:36.947660 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.947580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09296b9-b36a-4945-88be-6f70490e6e8c-tls-certs\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.947806 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.947669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d09296b9-b36a-4945-88be-6f70490e6e8c-tmp\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.947806 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.947713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w969c\" (UniqueName: \"kubernetes.io/projected/d09296b9-b36a-4945-88be-6f70490e6e8c-kube-api-access-w969c\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.950355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.950327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d09296b9-b36a-4945-88be-6f70490e6e8c-tmp\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.950869 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.950848 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09296b9-b36a-4945-88be-6f70490e6e8c-tls-certs\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:36.962503 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:36.962472 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w969c\" (UniqueName: \"kubernetes.io/projected/d09296b9-b36a-4945-88be-6f70490e6e8c-kube-api-access-w969c\") pod \"kube-auth-proxy-799fccbb74-8zwxm\" (UID: \"d09296b9-b36a-4945-88be-6f70490e6e8c\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:37.001989 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.001950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" Jul 09 14:18:37.048925 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.048705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-cert\") pod \"odh-model-controller-7998c67dc4-fk49v\" (UID: \"aa36c248-3d76-4fe6-b2ce-a237aac91a2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.048925 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.048838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km7md\" (UniqueName: \"kubernetes.io/projected/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-kube-api-access-km7md\") pod \"odh-model-controller-7998c67dc4-fk49v\" (UID: \"aa36c248-3d76-4fe6-b2ce-a237aac91a2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.150246 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.150180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km7md\" (UniqueName: \"kubernetes.io/projected/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-kube-api-access-km7md\") pod \"odh-model-controller-7998c67dc4-fk49v\" (UID: \"aa36c248-3d76-4fe6-b2ce-a237aac91a2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.150417 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.150289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-cert\") pod \"odh-model-controller-7998c67dc4-fk49v\" (UID: \"aa36c248-3d76-4fe6-b2ce-a237aac91a2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.150417 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:18:37.150399 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 09 14:18:37.150490 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:18:37.150462 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-cert podName:aa36c248-3d76-4fe6-b2ce-a237aac91a2b nodeName:}" failed. No retries permitted until 2026-07-09 14:18:37.650444399 +0000 UTC m=+284.339373630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-cert") pod "odh-model-controller-7998c67dc4-fk49v" (UID: "aa36c248-3d76-4fe6-b2ce-a237aac91a2b") : secret "odh-model-controller-webhook-cert" not found Jul 09 14:18:37.155983 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.155955 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm"] Jul 09 14:18:37.159596 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:18:37.159562 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09296b9_b36a_4945_88be_6f70490e6e8c.slice/crio-11b07125a76106337ee205189b665ae7d032487a07ec5250c804a00852e40e0a WatchSource:0}: Error finding container 11b07125a76106337ee205189b665ae7d032487a07ec5250c804a00852e40e0a: Status 404 returned error can't find the container with id 11b07125a76106337ee205189b665ae7d032487a07ec5250c804a00852e40e0a Jul 09 14:18:37.161619 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.161598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km7md\" (UniqueName: \"kubernetes.io/projected/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-kube-api-access-km7md\") pod \"odh-model-controller-7998c67dc4-fk49v\" (UID: \"aa36c248-3d76-4fe6-b2ce-a237aac91a2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.532645 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.532609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" event={"ID":"d09296b9-b36a-4945-88be-6f70490e6e8c","Type":"ContainerStarted","Data":"11b07125a76106337ee205189b665ae7d032487a07ec5250c804a00852e40e0a"} Jul 09 14:18:37.654368 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.654328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-cert\") pod \"odh-model-controller-7998c67dc4-fk49v\" (UID: \"aa36c248-3d76-4fe6-b2ce-a237aac91a2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.656720 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.656697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa36c248-3d76-4fe6-b2ce-a237aac91a2b-cert\") pod \"odh-model-controller-7998c67dc4-fk49v\" (UID: \"aa36c248-3d76-4fe6-b2ce-a237aac91a2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.756715 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.756678 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k2bnn"] Jul 09 14:18:37.761276 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.761245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:37.764680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.764659 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 09 14:18:37.765004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.764986 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nrbt8\"" Jul 09 14:18:37.787244 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.787165 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k2bnn"] Jul 09 14:18:37.794706 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.794685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:37.856365 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.855637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69b456e-1e0c-4b05-9a68-1953513ba62f-tls-certs\") pod \"model-serving-api-54d89bd5fc-k2bnn\" (UID: \"f69b456e-1e0c-4b05-9a68-1953513ba62f\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:37.856365 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.855801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqrw\" (UniqueName: \"kubernetes.io/projected/f69b456e-1e0c-4b05-9a68-1953513ba62f-kube-api-access-2vqrw\") pod \"model-serving-api-54d89bd5fc-k2bnn\" (UID: \"f69b456e-1e0c-4b05-9a68-1953513ba62f\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:37.957261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.952983 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fk49v"] Jul 09 14:18:37.957261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.956918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69b456e-1e0c-4b05-9a68-1953513ba62f-tls-certs\") pod \"model-serving-api-54d89bd5fc-k2bnn\" (UID: \"f69b456e-1e0c-4b05-9a68-1953513ba62f\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:37.957261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.956973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqrw\" (UniqueName: \"kubernetes.io/projected/f69b456e-1e0c-4b05-9a68-1953513ba62f-kube-api-access-2vqrw\") pod \"model-serving-api-54d89bd5fc-k2bnn\" (UID: \"f69b456e-1e0c-4b05-9a68-1953513ba62f\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:37.960668 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:18:37.957662 2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 09 14:18:37.960668 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:18:37.957788 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69b456e-1e0c-4b05-9a68-1953513ba62f-tls-certs podName:f69b456e-1e0c-4b05-9a68-1953513ba62f nodeName:}" failed. No retries permitted until 2026-07-09 14:18:38.457764149 +0000 UTC m=+285.146693381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f69b456e-1e0c-4b05-9a68-1953513ba62f-tls-certs") pod "model-serving-api-54d89bd5fc-k2bnn" (UID: "f69b456e-1e0c-4b05-9a68-1953513ba62f") : secret "model-serving-api-tls" not found Jul 09 14:18:37.974204 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:37.974169 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqrw\" (UniqueName: \"kubernetes.io/projected/f69b456e-1e0c-4b05-9a68-1953513ba62f-kube-api-access-2vqrw\") pod \"model-serving-api-54d89bd5fc-k2bnn\" (UID: \"f69b456e-1e0c-4b05-9a68-1953513ba62f\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:38.306989 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.306946 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-t484f"] Jul 09 14:18:38.310530 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.310504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.316152 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.316129 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 14:18:38.316306 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.316183 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 14:18:38.316702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.316678 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 14:18:38.317045 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.316700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 14:18:38.317045 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.316722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m9lsk\"" Jul 09 14:18:38.331120 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.331085 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-t484f"] Jul 09 14:18:38.360925 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.360891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.361088 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.360946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.361088 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.361022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-local-certs\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.361088 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.361047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-token\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.361088 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.361070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlzq5\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-kube-api-access-mlzq5\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.361249 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.361098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.361249 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.361212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-cacerts\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.462239 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-cacerts\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.462423 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.462479 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69b456e-1e0c-4b05-9a68-1953513ba62f-tls-certs\") pod \"model-serving-api-54d89bd5fc-k2bnn\" (UID: \"f69b456e-1e0c-4b05-9a68-1953513ba62f\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:38.462479 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.462585 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-local-certs\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.462761 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-token\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.462883 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzq5\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-kube-api-access-mlzq5\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.462883 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.462811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.463501 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.463476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.464717 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.464690 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-cacerts\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.464861 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.464842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.464920 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.464845 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.465099 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.465080 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-local-certs\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.465293 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.465280 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69b456e-1e0c-4b05-9a68-1953513ba62f-tls-certs\") pod \"model-serving-api-54d89bd5fc-k2bnn\" (UID: \"f69b456e-1e0c-4b05-9a68-1953513ba62f\") " pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:38.471327 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.471305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-token\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.471421 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.471401 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzq5\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-kube-api-access-mlzq5\") pod \"istiod-openshift-gateway-7574459656-t484f\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.538290 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.538252 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" event={"ID":"aa36c248-3d76-4fe6-b2ce-a237aac91a2b","Type":"ContainerStarted","Data":"e830f7f4aab5fee72510ba9ffc444dd11a96c66d5411c2cc19fe21deda5c3d8f"} Jul 09 14:18:38.622945 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.622857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:38.672104 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.671833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:38.773039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.773007 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-t484f"] Jul 09 14:18:38.829998 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:38.829975 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k2bnn"] Jul 09 14:18:38.832062 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:18:38.832034 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69b456e_1e0c_4b05_9a68_1953513ba62f.slice/crio-52a34610fce5e1a61a9b0e84aa33f11bfcc3ee28fa3994a4a09404ca77f9d08f WatchSource:0}: Error finding container 52a34610fce5e1a61a9b0e84aa33f11bfcc3ee28fa3994a4a09404ca77f9d08f: Status 404 returned error can't find the container with id 52a34610fce5e1a61a9b0e84aa33f11bfcc3ee28fa3994a4a09404ca77f9d08f Jul 09 14:18:39.545660 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:39.545576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" event={"ID":"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b","Type":"ContainerStarted","Data":"c540a6a15fd6107bc9bca2091d2807abd1907d264357be0a1a056e223adac968"} Jul 09 14:18:39.547069 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:39.547031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" event={"ID":"f69b456e-1e0c-4b05-9a68-1953513ba62f","Type":"ContainerStarted","Data":"52a34610fce5e1a61a9b0e84aa33f11bfcc3ee28fa3994a4a09404ca77f9d08f"} Jul 09 14:18:43.459054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.459010 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:18:43.459369 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.459096 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:18:43.573009 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.572902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" event={"ID":"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b","Type":"ContainerStarted","Data":"7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461"} Jul 09 14:18:43.573009 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.572976 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:43.575148 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.575113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" event={"ID":"d09296b9-b36a-4945-88be-6f70490e6e8c","Type":"ContainerStarted","Data":"37d31dcc1f6fb3e753b020b17f699c161ba95998aad56e19a3e122624d8622bb"} Jul 09 14:18:43.577359 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.577260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" event={"ID":"aa36c248-3d76-4fe6-b2ce-a237aac91a2b","Type":"ContainerStarted","Data":"ef285b74aba6d28547eec9e587a1e4c95004e79e18760994fb52e20a6bdba93d"} Jul 09 14:18:43.577359 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.577323 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:43.579392 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.579365 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" event={"ID":"f69b456e-1e0c-4b05-9a68-1953513ba62f","Type":"ContainerStarted","Data":"e09983291d6be2734986ba4697fc59730f56b29cc0d5db8852cee61d9ae31490"} Jul 09 14:18:43.579531 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.579512 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:18:43.603098 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.603046 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" podStartSLOduration=0.923646003 podStartE2EDuration="5.60302991s" totalImagesPullingTime="4.679383907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:43.601473183 +0000 UTC m=+290.290402423" watchObservedRunningTime="2026-07-09 14:18:43.60302991 +0000 UTC m=+290.291959150" Jul 09 14:18:43.623043 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.622991 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" podStartSLOduration=2.115590924 podStartE2EDuration="7.622974826s" totalImagesPullingTime="5.507383902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:43.622299241 +0000 UTC m=+290.311228508" watchObservedRunningTime="2026-07-09 14:18:43.622974826 +0000 UTC m=+290.311904090" Jul 09 14:18:43.643501 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.643365 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" podStartSLOduration=2.025675892 podStartE2EDuration="6.643348386s" totalImagesPullingTime="4.617672494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:43.641805028 +0000 UTC m=+290.330734268" watchObservedRunningTime="2026-07-09 14:18:43.643348386 +0000 UTC m=+290.332277624" Jul 09 14:18:43.664547 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:43.664460 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-799fccbb74-8zwxm" podStartSLOduration=1.435850761 podStartE2EDuration="7.664421939s" totalImagesPullingTime="6.228571178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:43.662090779 +0000 UTC m=+290.351020013" watchObservedRunningTime="2026-07-09 14:18:43.664421939 +0000 UTC m=+290.353351179" Jul 09 14:18:44.586185 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:44.586150 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-t484f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 14:18:44.586628 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:44.586204 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" podUID="88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 14:18:47.585996 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.585956 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:47.663515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.663116 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv"] Jul 09 14:18:47.670034 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.670004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:47.674121 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.674098 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jq8rm\"" Jul 09 14:18:47.674432 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.674401 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 14:18:47.680113 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.680084 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv"] Jul 09 14:18:47.752917 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.752868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73951f2a-1722-4654-85f3-908da3a1a9ba-cert\") pod \"kserve-controller-manager-d6f4b65b8-kvzvv\" (UID: \"73951f2a-1722-4654-85f3-908da3a1a9ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:47.753123 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.752954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n92wk\" (UniqueName: \"kubernetes.io/projected/73951f2a-1722-4654-85f3-908da3a1a9ba-kube-api-access-n92wk\") pod \"kserve-controller-manager-d6f4b65b8-kvzvv\" (UID: \"73951f2a-1722-4654-85f3-908da3a1a9ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:47.854046 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.853944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73951f2a-1722-4654-85f3-908da3a1a9ba-cert\") pod \"kserve-controller-manager-d6f4b65b8-kvzvv\" (UID: \"73951f2a-1722-4654-85f3-908da3a1a9ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:47.854046 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.854003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n92wk\" (UniqueName: \"kubernetes.io/projected/73951f2a-1722-4654-85f3-908da3a1a9ba-kube-api-access-n92wk\") pod \"kserve-controller-manager-d6f4b65b8-kvzvv\" (UID: \"73951f2a-1722-4654-85f3-908da3a1a9ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:47.856451 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.856426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73951f2a-1722-4654-85f3-908da3a1a9ba-cert\") pod \"kserve-controller-manager-d6f4b65b8-kvzvv\" (UID: \"73951f2a-1722-4654-85f3-908da3a1a9ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:47.876812 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.876782 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n92wk\" (UniqueName: \"kubernetes.io/projected/73951f2a-1722-4654-85f3-908da3a1a9ba-kube-api-access-n92wk\") pod \"kserve-controller-manager-d6f4b65b8-kvzvv\" (UID: \"73951f2a-1722-4654-85f3-908da3a1a9ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:47.982004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:47.981963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:48.126802 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:48.126768 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv"] Jul 09 14:18:48.128940 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:18:48.128912 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73951f2a_1722_4654_85f3_908da3a1a9ba.slice/crio-ba8910acf6c4dd5c12bbf29ad539fb3994ae991421d9f9f0c64ca7176f33547c WatchSource:0}: Error finding container ba8910acf6c4dd5c12bbf29ad539fb3994ae991421d9f9f0c64ca7176f33547c: Status 404 returned error can't find the container with id ba8910acf6c4dd5c12bbf29ad539fb3994ae991421d9f9f0c64ca7176f33547c Jul 09 14:18:48.605816 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:48.605779 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" event={"ID":"73951f2a-1722-4654-85f3-908da3a1a9ba","Type":"ContainerStarted","Data":"ba8910acf6c4dd5c12bbf29ad539fb3994ae991421d9f9f0c64ca7176f33547c"} Jul 09 14:18:51.623746 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:51.623708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" event={"ID":"73951f2a-1722-4654-85f3-908da3a1a9ba","Type":"ContainerStarted","Data":"38e5274052c525e654f1549c7096b183b8106e57509aa2842bb7f5669f27ee19"} Jul 09 14:18:51.624257 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:51.623841 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:18:51.655994 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:51.655939 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" podStartSLOduration=1.641687168 podStartE2EDuration="4.655923131s" totalImagesPullingTime="3.014235963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:51.653278456 +0000 UTC m=+298.342207717" watchObservedRunningTime="2026-07-09 14:18:51.655923131 +0000 UTC m=+298.344852369" Jul 09 14:18:53.270453 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.270418 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-t484f"] Jul 09 14:18:53.270913 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.270602 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" podUID="88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" containerName="discovery" containerID="cri-o://7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461" gracePeriod=30 Jul 09 14:18:53.520320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.520292 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:53.610745 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.610714 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-local-certs\") pod \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " Jul 09 14:18:53.610956 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.610778 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-dns-cert\") pod \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " Jul 09 14:18:53.610956 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.610833 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlzq5\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-kube-api-access-mlzq5\") pod \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " Jul 09 14:18:53.610956 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.610870 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-ca-configmap\") pod \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " Jul 09 14:18:53.610956 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.610906 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-token\") pod \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " Jul 09 14:18:53.610956 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.610931 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-cacerts\") pod \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " Jul 09 14:18:53.611218 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.610995 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-kubeconfig\") pod \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\" (UID: \"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b\") " Jul 09 14:18:53.611374 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.611343 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" (UID: "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:18:53.613486 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.613432 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" (UID: "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:18:53.613601 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.613545 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-local-certs" (OuterVolumeSpecName: "local-certs") pod "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" (UID: "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:18:53.613601 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.613573 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-token" (OuterVolumeSpecName: "istio-token") pod "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" (UID: "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:18:53.613708 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.613654 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" (UID: "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:18:53.613783 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.613757 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-kube-api-access-mlzq5" (OuterVolumeSpecName: "kube-api-access-mlzq5") pod "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" (UID: "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b"). InnerVolumeSpecName "kube-api-access-mlzq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:18:53.614376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.614352 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-cacerts" (OuterVolumeSpecName: "cacerts") pod "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" (UID: "88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:18:53.635206 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.635176 2575 generic.go:356] "Generic (PLEG): container finished" podID="88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" containerID="7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461" exitCode=0 Jul 09 14:18:53.635345 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.635241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" Jul 09 14:18:53.635345 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.635257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" event={"ID":"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b","Type":"ContainerDied","Data":"7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461"} Jul 09 14:18:53.635345 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.635336 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-t484f" event={"ID":"88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b","Type":"ContainerDied","Data":"c540a6a15fd6107bc9bca2091d2807abd1907d264357be0a1a056e223adac968"} Jul 09 14:18:53.635515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.635366 2575 scope.go:117] "RemoveContainer" containerID="7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461" Jul 09 14:18:53.644730 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.644710 2575 scope.go:117] "RemoveContainer" containerID="7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461" Jul 09 14:18:53.645003 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:18:53.644985 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461\": container with ID starting with 7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461 not found: ID does not exist" containerID="7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461" Jul 09 14:18:53.645064 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.645013 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461"} err="failed to get container status \"7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461\": rpc error: code = NotFound desc = could not find container \"7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461\": container with ID starting with 7421bc994d95aea3d9435320081042cc65ac4a0f458f2978b04dd02c052de461 not found: ID does not exist" Jul 09 14:18:53.661099 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.661074 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-t484f"] Jul 09 14:18:53.664853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.664831 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-t484f"] Jul 09 14:18:53.711811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.711781 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlzq5\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-kube-api-access-mlzq5\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:18:53.711811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.711808 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-ca-configmap\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:18:53.711811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.711818 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:18:53.712051 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.711827 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-cacerts\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:18:53.712051 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.711835 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-kubeconfig\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:18:53.712051 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.711842 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-local-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:18:53.712051 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.711850 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b-istio-csr-dns-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:18:53.755649 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.755624 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:18:53.756271 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.756246 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:18:53.766795 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.766772 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 14:18:53.888704 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:53.888625 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" path="/var/lib/kubelet/pods/88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b/volumes" Jul 09 14:18:54.586027 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:54.585998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fk49v" Jul 09 14:18:54.588418 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:18:54.588168 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-k2bnn" Jul 09 14:19:22.635148 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:22.635103 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kvzvv" Jul 09 14:19:37.425275 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.425238 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-96dn6"] Jul 09 14:19:37.425650 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.425582 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" containerName="discovery" Jul 09 14:19:37.425650 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.425592 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" containerName="discovery" Jul 09 14:19:37.425750 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.425658 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88841f7a-7b4a-433f-b4dd-a6ec6dd1e79b" containerName="discovery" Jul 09 14:19:37.431686 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.431659 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" Jul 09 14:19:37.434329 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.434300 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 14:19:37.434487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.434300 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 14:19:37.435293 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.435271 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-k82tv\"" Jul 09 14:19:37.440964 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.440935 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-96dn6"] Jul 09 14:19:37.475337 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.475295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtpq\" (UniqueName: \"kubernetes.io/projected/1bb4b89b-1756-429d-a2a4-ae73b2dbfb69-kube-api-access-gmtpq\") pod \"kuadrant-operator-catalog-96dn6\" (UID: \"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69\") " pod="kuadrant-system/kuadrant-operator-catalog-96dn6" Jul 09 14:19:37.576785 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.576754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtpq\" (UniqueName: \"kubernetes.io/projected/1bb4b89b-1756-429d-a2a4-ae73b2dbfb69-kube-api-access-gmtpq\") pod \"kuadrant-operator-catalog-96dn6\" (UID: \"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69\") " pod="kuadrant-system/kuadrant-operator-catalog-96dn6" Jul 09 14:19:37.585598 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.585564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtpq\" (UniqueName: \"kubernetes.io/projected/1bb4b89b-1756-429d-a2a4-ae73b2dbfb69-kube-api-access-gmtpq\") pod \"kuadrant-operator-catalog-96dn6\" (UID: \"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69\") " pod="kuadrant-system/kuadrant-operator-catalog-96dn6" Jul 09 14:19:37.743615 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.743518 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" Jul 09 14:19:37.786292 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.786259 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-96dn6"] Jul 09 14:19:37.887528 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.887504 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-96dn6"] Jul 09 14:19:37.889552 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:19:37.889517 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb4b89b_1756_429d_a2a4_ae73b2dbfb69.slice/crio-d41202e566a68c24a20ad9c81196ab5a971a5aa74f409f012b4544852aae3fd3 WatchSource:0}: Error finding container d41202e566a68c24a20ad9c81196ab5a971a5aa74f409f012b4544852aae3fd3: Status 404 returned error can't find the container with id d41202e566a68c24a20ad9c81196ab5a971a5aa74f409f012b4544852aae3fd3 Jul 09 14:19:37.890753 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.890735 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:19:37.996479 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:37.996406 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zpvfx"] Jul 09 14:19:38.001781 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.001761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:38.008343 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.008314 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zpvfx"] Jul 09 14:19:38.081609 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.081565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhkk\" (UniqueName: \"kubernetes.io/projected/7ac96cb2-4d04-4eeb-a2f9-276d62db5e31-kube-api-access-7fhkk\") pod \"kuadrant-operator-catalog-zpvfx\" (UID: \"7ac96cb2-4d04-4eeb-a2f9-276d62db5e31\") " pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:38.182558 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.182519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhkk\" (UniqueName: \"kubernetes.io/projected/7ac96cb2-4d04-4eeb-a2f9-276d62db5e31-kube-api-access-7fhkk\") pod \"kuadrant-operator-catalog-zpvfx\" (UID: \"7ac96cb2-4d04-4eeb-a2f9-276d62db5e31\") " pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:38.199285 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.199258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhkk\" (UniqueName: \"kubernetes.io/projected/7ac96cb2-4d04-4eeb-a2f9-276d62db5e31-kube-api-access-7fhkk\") pod \"kuadrant-operator-catalog-zpvfx\" (UID: \"7ac96cb2-4d04-4eeb-a2f9-276d62db5e31\") " pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:38.312469 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.312434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:38.447740 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.447717 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zpvfx"] Jul 09 14:19:38.449435 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:19:38.449408 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac96cb2_4d04_4eeb_a2f9_276d62db5e31.slice/crio-393a392aae6c451f1f503740c7d7f1067d3a61960d2da8fd614e5f228eb1268a WatchSource:0}: Error finding container 393a392aae6c451f1f503740c7d7f1067d3a61960d2da8fd614e5f228eb1268a: Status 404 returned error can't find the container with id 393a392aae6c451f1f503740c7d7f1067d3a61960d2da8fd614e5f228eb1268a Jul 09 14:19:38.834539 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.834499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" event={"ID":"7ac96cb2-4d04-4eeb-a2f9-276d62db5e31","Type":"ContainerStarted","Data":"393a392aae6c451f1f503740c7d7f1067d3a61960d2da8fd614e5f228eb1268a"} Jul 09 14:19:38.836047 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:38.836006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" event={"ID":"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69","Type":"ContainerStarted","Data":"d41202e566a68c24a20ad9c81196ab5a971a5aa74f409f012b4544852aae3fd3"} Jul 09 14:19:40.848352 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:40.848314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" event={"ID":"7ac96cb2-4d04-4eeb-a2f9-276d62db5e31","Type":"ContainerStarted","Data":"3e43e200a60af4443647c0bd32c0b8cea2a2993a8469ce5abff382894ccd6009"} Jul 09 14:19:40.849702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:40.849674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" event={"ID":"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69","Type":"ContainerStarted","Data":"d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe"} Jul 09 14:19:40.849782 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:40.849741 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" podUID="1bb4b89b-1756-429d-a2a4-ae73b2dbfb69" containerName="registry-server" containerID="cri-o://d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe" gracePeriod=2 Jul 09 14:19:40.865791 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:40.865732 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" podStartSLOduration=2.197611173 podStartE2EDuration="3.865713123s" totalImagesPullingTime="1.66810195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:19:40.863938775 +0000 UTC m=+347.552868037" watchObservedRunningTime="2026-07-09 14:19:40.865713123 +0000 UTC m=+347.554642364" Jul 09 14:19:40.885498 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:40.885447 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" podStartSLOduration=1.653466169 podStartE2EDuration="3.885430462s" totalImagesPullingTime="2.231964293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:19:40.883815898 +0000 UTC m=+347.572745137" watchObservedRunningTime="2026-07-09 14:19:40.885430462 +0000 UTC m=+347.574359702" Jul 09 14:19:41.105964 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.105894 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" Jul 09 14:19:41.210934 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.210882 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmtpq\" (UniqueName: \"kubernetes.io/projected/1bb4b89b-1756-429d-a2a4-ae73b2dbfb69-kube-api-access-gmtpq\") pod \"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69\" (UID: \"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69\") " Jul 09 14:19:41.213082 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.213051 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb4b89b-1756-429d-a2a4-ae73b2dbfb69-kube-api-access-gmtpq" (OuterVolumeSpecName: "kube-api-access-gmtpq") pod "1bb4b89b-1756-429d-a2a4-ae73b2dbfb69" (UID: "1bb4b89b-1756-429d-a2a4-ae73b2dbfb69"). InnerVolumeSpecName "kube-api-access-gmtpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:19:41.312425 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.312373 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmtpq\" (UniqueName: \"kubernetes.io/projected/1bb4b89b-1756-429d-a2a4-ae73b2dbfb69-kube-api-access-gmtpq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:19:41.856033 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.855995 2575 generic.go:356] "Generic (PLEG): container finished" podID="1bb4b89b-1756-429d-a2a4-ae73b2dbfb69" containerID="d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe" exitCode=0 Jul 09 14:19:41.856440 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.856038 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" Jul 09 14:19:41.856440 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.856077 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" event={"ID":"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69","Type":"ContainerDied","Data":"d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe"} Jul 09 14:19:41.856440 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.856144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-96dn6" event={"ID":"1bb4b89b-1756-429d-a2a4-ae73b2dbfb69","Type":"ContainerDied","Data":"d41202e566a68c24a20ad9c81196ab5a971a5aa74f409f012b4544852aae3fd3"} Jul 09 14:19:41.856440 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.856165 2575 scope.go:117] "RemoveContainer" containerID="d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe" Jul 09 14:19:41.866084 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.866069 2575 scope.go:117] "RemoveContainer" containerID="d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe" Jul 09 14:19:41.866359 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:19:41.866343 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe\": container with ID starting with d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe not found: ID does not exist" containerID="d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe" Jul 09 14:19:41.866406 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.866369 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe"} err="failed to get container status \"d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe\": rpc error: code = NotFound desc = could not find container \"d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe\": container with ID starting with d39e3d7e000d13fb10370064f8b79626e45d8aa64f00abebb1e84bd9f63afffe not found: ID does not exist" Jul 09 14:19:41.879048 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.879024 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-96dn6"] Jul 09 14:19:41.886483 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:41.886459 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-96dn6"] Jul 09 14:19:43.885356 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:43.885322 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bb4b89b-1756-429d-a2a4-ae73b2dbfb69" path="/var/lib/kubelet/pods/1bb4b89b-1756-429d-a2a4-ae73b2dbfb69/volumes" Jul 09 14:19:48.313052 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:48.312962 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:48.313052 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:48.313022 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:48.335842 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:48.335814 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:19:48.909457 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:19:48.909430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zpvfx" Jul 09 14:20:10.294965 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.294930 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5"] Jul 09 14:20:10.295552 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.295457 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb4b89b-1756-429d-a2a4-ae73b2dbfb69" containerName="registry-server" Jul 09 14:20:10.295552 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.295477 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb4b89b-1756-429d-a2a4-ae73b2dbfb69" containerName="registry-server" Jul 09 14:20:10.295670 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.295592 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb4b89b-1756-429d-a2a4-ae73b2dbfb69" containerName="registry-server" Jul 09 14:20:10.303352 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.303328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" Jul 09 14:20:10.306057 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.306028 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dg6ch\"" Jul 09 14:20:10.313714 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.313692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5"] Jul 09 14:20:10.466420 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.466379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvq8\" (UniqueName: \"kubernetes.io/projected/3edf0e79-eea6-4861-9d8e-643621d944dc-kube-api-access-7xvq8\") pod \"authorino-operator-75cc8d5cb7-xxpw5\" (UID: \"3edf0e79-eea6-4861-9d8e-643621d944dc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" Jul 09 14:20:10.566968 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.566883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvq8\" (UniqueName: \"kubernetes.io/projected/3edf0e79-eea6-4861-9d8e-643621d944dc-kube-api-access-7xvq8\") pod \"authorino-operator-75cc8d5cb7-xxpw5\" (UID: \"3edf0e79-eea6-4861-9d8e-643621d944dc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" Jul 09 14:20:10.584737 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.584705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvq8\" (UniqueName: \"kubernetes.io/projected/3edf0e79-eea6-4861-9d8e-643621d944dc-kube-api-access-7xvq8\") pod \"authorino-operator-75cc8d5cb7-xxpw5\" (UID: \"3edf0e79-eea6-4861-9d8e-643621d944dc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" Jul 09 14:20:10.613358 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.613329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" Jul 09 14:20:10.761153 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.761116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5"] Jul 09 14:20:10.767343 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:20:10.767311 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edf0e79_eea6_4861_9d8e_643621d944dc.slice/crio-d7e40c87343f71a3efb8a9ab62c820a2280896efd5e4f1e6863bbd163f20e4d9 WatchSource:0}: Error finding container d7e40c87343f71a3efb8a9ab62c820a2280896efd5e4f1e6863bbd163f20e4d9: Status 404 returned error can't find the container with id d7e40c87343f71a3efb8a9ab62c820a2280896efd5e4f1e6863bbd163f20e4d9 Jul 09 14:20:10.986161 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:10.986071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" event={"ID":"3edf0e79-eea6-4861-9d8e-643621d944dc","Type":"ContainerStarted","Data":"d7e40c87343f71a3efb8a9ab62c820a2280896efd5e4f1e6863bbd163f20e4d9"} Jul 09 14:20:13.127441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.127399 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5"] Jul 09 14:20:13.131471 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.131445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:13.134623 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.134578 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vt6rf\"" Jul 09 14:20:13.159066 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.159029 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5"] Jul 09 14:20:13.191043 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.191009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7kl\" (UniqueName: \"kubernetes.io/projected/35820e70-2e70-4f9f-9a9e-12fe464ffe2b-kube-api-access-pb7kl\") pod \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" (UID: \"35820e70-2e70-4f9f-9a9e-12fe464ffe2b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:13.291768 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.291741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7kl\" (UniqueName: \"kubernetes.io/projected/35820e70-2e70-4f9f-9a9e-12fe464ffe2b-kube-api-access-pb7kl\") pod \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" (UID: \"35820e70-2e70-4f9f-9a9e-12fe464ffe2b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:13.312289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.312213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7kl\" (UniqueName: \"kubernetes.io/projected/35820e70-2e70-4f9f-9a9e-12fe464ffe2b-kube-api-access-pb7kl\") pod \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" (UID: \"35820e70-2e70-4f9f-9a9e-12fe464ffe2b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:13.445500 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.445418 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:13.593268 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:13.593244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5"] Jul 09 14:20:13.596094 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:20:13.596067 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35820e70_2e70_4f9f_9a9e_12fe464ffe2b.slice/crio-c2ce36614f75d75827b6d4aa5e0f6f2e0207c8444db2dd0e4a3a8d36c8edabc8 WatchSource:0}: Error finding container c2ce36614f75d75827b6d4aa5e0f6f2e0207c8444db2dd0e4a3a8d36c8edabc8: Status 404 returned error can't find the container with id c2ce36614f75d75827b6d4aa5e0f6f2e0207c8444db2dd0e4a3a8d36c8edabc8 Jul 09 14:20:14.003471 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:14.003430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" event={"ID":"3edf0e79-eea6-4861-9d8e-643621d944dc","Type":"ContainerStarted","Data":"8730518eda0af70140b1ff7052c107ee341ed577519ae7d03c8f979be5e3301e"} Jul 09 14:20:14.003671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:14.003576 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" Jul 09 14:20:14.004639 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:14.004615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" event={"ID":"35820e70-2e70-4f9f-9a9e-12fe464ffe2b","Type":"ContainerStarted","Data":"c2ce36614f75d75827b6d4aa5e0f6f2e0207c8444db2dd0e4a3a8d36c8edabc8"} Jul 09 14:20:14.029008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:14.028952 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" podStartSLOduration=1.543295171 podStartE2EDuration="4.028891119s" totalImagesPullingTime="2.485595948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:14.028152665 +0000 UTC m=+380.717081933" watchObservedRunningTime="2026-07-09 14:20:14.028891119 +0000 UTC m=+380.717820359" Jul 09 14:20:16.015672 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:16.015567 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" event={"ID":"35820e70-2e70-4f9f-9a9e-12fe464ffe2b","Type":"ContainerStarted","Data":"8d05c83c1d76c17c9b098a115dcd8a243b10c4fa5cb720f03d05b9390e708d49"} Jul 09 14:20:16.015672 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:16.015647 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:16.037925 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:16.037874 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" podStartSLOduration=1.014112893 podStartE2EDuration="3.037857591s" totalImagesPullingTime="2.023744698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:16.035762317 +0000 UTC m=+382.724691557" watchObservedRunningTime="2026-07-09 14:20:16.037857591 +0000 UTC m=+382.726786829" Jul 09 14:20:16.866092 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:16.866051 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9"] Jul 09 14:20:16.869417 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:16.869399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:16.872485 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:16.872463 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-csmzx\"" Jul 09 14:20:16.891711 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:16.891673 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9"] Jul 09 14:20:17.017160 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.017127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac733db-3d32-48e9-bb9f-19749da5bdbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8wqt9\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:17.017160 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.017161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kklfn\" (UniqueName: \"kubernetes.io/projected/cac733db-3d32-48e9-bb9f-19749da5bdbe-kube-api-access-kklfn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8wqt9\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:17.117681 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.117590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac733db-3d32-48e9-bb9f-19749da5bdbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8wqt9\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:17.117681 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.117635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kklfn\" (UniqueName: \"kubernetes.io/projected/cac733db-3d32-48e9-bb9f-19749da5bdbe-kube-api-access-kklfn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8wqt9\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:17.117934 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.117912 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac733db-3d32-48e9-bb9f-19749da5bdbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8wqt9\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:17.129741 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.129714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kklfn\" (UniqueName: \"kubernetes.io/projected/cac733db-3d32-48e9-bb9f-19749da5bdbe-kube-api-access-kklfn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8wqt9\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:17.179320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.179284 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:17.326718 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:17.326691 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9"] Jul 09 14:20:17.328462 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:20:17.328433 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac733db_3d32_48e9_bb9f_19749da5bdbe.slice/crio-33d46efe2cf01ac08a32c8625ee57b7cc7f0c9ccf3a47bba12312f8c2326479f WatchSource:0}: Error finding container 33d46efe2cf01ac08a32c8625ee57b7cc7f0c9ccf3a47bba12312f8c2326479f: Status 404 returned error can't find the container with id 33d46efe2cf01ac08a32c8625ee57b7cc7f0c9ccf3a47bba12312f8c2326479f Jul 09 14:20:18.027022 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:18.026974 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" event={"ID":"cac733db-3d32-48e9-bb9f-19749da5bdbe","Type":"ContainerStarted","Data":"33d46efe2cf01ac08a32c8625ee57b7cc7f0c9ccf3a47bba12312f8c2326479f"} Jul 09 14:20:23.053617 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:23.053579 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" event={"ID":"cac733db-3d32-48e9-bb9f-19749da5bdbe","Type":"ContainerStarted","Data":"c6067df9211965243edbe78dad054098c2768a6c1fc2ba7f8ac6ec8ed905ba7a"} Jul 09 14:20:23.054075 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:23.053663 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:23.078448 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:23.078391 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" podStartSLOduration=1.536842197 podStartE2EDuration="7.078373236s" totalImagesPullingTime="5.541531039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:23.076316741 +0000 UTC m=+389.765245999" watchObservedRunningTime="2026-07-09 14:20:23.078373236 +0000 UTC m=+389.767302477" Jul 09 14:20:25.011760 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:25.011723 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xxpw5" Jul 09 14:20:27.022085 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:27.022054 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:34.061211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:34.061182 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:35.223478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.223448 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq"] Jul 09 14:20:35.227109 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.227092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:35.242406 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.242376 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq"] Jul 09 14:20:35.284321 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.284289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rs7\" (UniqueName: \"kubernetes.io/projected/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-kube-api-access-z8rs7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:35.284448 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.284345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:35.342263 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.342217 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq"] Jul 09 14:20:35.342498 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:35.342476 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-z8rs7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" Jul 09 14:20:35.385468 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.385421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rs7\" (UniqueName: \"kubernetes.io/projected/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-kube-api-access-z8rs7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:35.385468 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.385480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:35.385847 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.385829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:35.391325 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.391294 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8"] Jul 09 14:20:35.394802 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.394786 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.397683 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.397660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rs7\" (UniqueName: \"kubernetes.io/projected/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-kube-api-access-z8rs7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:35.411354 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.411329 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8"] Jul 09 14:20:35.486433 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.486343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ebf175c-e55f-446e-853a-f10400244c1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.486433 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.486383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwm6r\" (UniqueName: \"kubernetes.io/projected/0ebf175c-e55f-446e-853a-f10400244c1d-kube-api-access-nwm6r\") pod \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.587108 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.587067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ebf175c-e55f-446e-853a-f10400244c1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.587108 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.587117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwm6r\" (UniqueName: \"kubernetes.io/projected/0ebf175c-e55f-446e-853a-f10400244c1d-kube-api-access-nwm6r\") pod \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.587583 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.587557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ebf175c-e55f-446e-853a-f10400244c1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.596268 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.596218 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwm6r\" (UniqueName: \"kubernetes.io/projected/0ebf175c-e55f-446e-853a-f10400244c1d-kube-api-access-nwm6r\") pod \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.713503 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.713463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:35.785306 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.785275 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8"] Jul 09 14:20:35.803374 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.803340 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8"] Jul 09 14:20:35.811525 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.811495 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54"] Jul 09 14:20:35.815879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.815853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:35.816025 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.816005 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq"] Jul 09 14:20:35.823779 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.823748 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq"] Jul 09 14:20:35.836756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.836731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54"] Jul 09 14:20:35.848165 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.848133 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9"] Jul 09 14:20:35.848395 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.848327 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" podUID="cac733db-3d32-48e9-bb9f-19749da5bdbe" containerName="manager" containerID="cri-o://c6067df9211965243edbe78dad054098c2768a6c1fc2ba7f8ac6ec8ed905ba7a" gracePeriod=2 Jul 09 14:20:35.869517 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.869487 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9"] Jul 09 14:20:35.887447 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:35.887407 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8wqt9\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="cac733db-3d32-48e9-bb9f-19749da5bdbe" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:35.890216 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.890189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n58s\" (UniqueName: \"kubernetes.io/projected/e6261c9b-3b32-42e7-bb73-984fbf424b9e-kube-api-access-6n58s\") pod \"kuadrant-operator-controller-manager-55467d9f7-g6c54\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:35.890384 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.890257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6261c9b-3b32-42e7-bb73-984fbf424b9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-g6c54\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:35.894825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.894626 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4"] Jul 09 14:20:35.895128 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.895107 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac733db-3d32-48e9-bb9f-19749da5bdbe" containerName="manager" Jul 09 14:20:35.895128 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.895126 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac733db-3d32-48e9-bb9f-19749da5bdbe" containerName="manager" Jul 09 14:20:35.895264 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.895183 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac733db-3d32-48e9-bb9f-19749da5bdbe" containerName="manager" Jul 09 14:20:35.898251 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.898209 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5"] Jul 09 14:20:35.898350 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.898335 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:35.898482 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.898455 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" containerName="manager" containerID="cri-o://8d05c83c1d76c17c9b098a115dcd8a243b10c4fa5cb720f03d05b9390e708d49" gracePeriod=2 Jul 09 14:20:35.918766 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.918740 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5"] Jul 09 14:20:35.937661 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:35.937618 2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"35820e70-2e70-4f9f-9a9e-12fe464ffe2b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-6fzd5\": pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:35.938278 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.938258 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4"] Jul 09 14:20:35.965906 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.965747 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx"] Jul 09 14:20:35.966380 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.966357 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" containerName="manager" Jul 09 14:20:35.966380 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.966381 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" containerName="manager" Jul 09 14:20:35.966546 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.966480 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" containerName="manager" Jul 09 14:20:35.969668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.969647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" Jul 09 14:20:35.991705 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.991674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n58s\" (UniqueName: \"kubernetes.io/projected/e6261c9b-3b32-42e7-bb73-984fbf424b9e-kube-api-access-6n58s\") pod \"kuadrant-operator-controller-manager-55467d9f7-g6c54\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:35.991860 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.991723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87e40348-9450-4eca-9438-8e04da9d7e80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x5wg4\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:35.991860 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.991774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6261c9b-3b32-42e7-bb73-984fbf424b9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-g6c54\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:35.992001 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.991866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8slg\" (UniqueName: \"kubernetes.io/projected/87e40348-9450-4eca-9438-8e04da9d7e80-kube-api-access-g8slg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x5wg4\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:35.992689 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.992279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6261c9b-3b32-42e7-bb73-984fbf424b9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-g6c54\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:35.995250 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:35.995215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx"] Jul 09 14:20:36.016780 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.016741 2575 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 09 14:20:36.016780 ip-10-0-141-45 kubenswrapper[2575]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-85cm8_kuadrant-system_0ebf175c-e55f-446e-853a-f10400244c1d_0(ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-85cm8 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773" Netns:"/var/run/netns/bf3a5bf3-7e8d-4d0d-8e62-a001e1a93156" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-85cm8;K8S_POD_INFRA_CONTAINER_ID=ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773;K8S_POD_UID=0ebf175c-e55f-446e-853a-f10400244c1d" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8/0ebf175c-e55f-446e-853a-f10400244c1d:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8 ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773 network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8 ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:29 [10.132.0.41/23] Jul 09 14:20:36.016780 ip-10-0-141-45 kubenswrapper[2575]: ' Jul 09 14:20:36.016780 ip-10-0-141-45 kubenswrapper[2575]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 09 14:20:36.016780 ip-10-0-141-45 kubenswrapper[2575]: > Jul 09 14:20:36.017048 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.016813 2575 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 09 14:20:36.017048 ip-10-0-141-45 kubenswrapper[2575]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-85cm8_kuadrant-system_0ebf175c-e55f-446e-853a-f10400244c1d_0(ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-85cm8 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773" Netns:"/var/run/netns/bf3a5bf3-7e8d-4d0d-8e62-a001e1a93156" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-85cm8;K8S_POD_INFRA_CONTAINER_ID=ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773;K8S_POD_UID=0ebf175c-e55f-446e-853a-f10400244c1d" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8/0ebf175c-e55f-446e-853a-f10400244c1d:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8 ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773 network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8 ccab559f1e64980344a332819dceaaf280ad3d53f3a66dd720aa9e81ed00f773 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:29 [10.132.0.41/23] Jul 09 14:20:36.017048 ip-10-0-141-45 kubenswrapper[2575]: ' Jul 09 14:20:36.017048 ip-10-0-141-45 kubenswrapper[2575]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 09 14:20:36.017048 ip-10-0-141-45 kubenswrapper[2575]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:36.021435 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.021412 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n58s\" (UniqueName: \"kubernetes.io/projected/e6261c9b-3b32-42e7-bb73-984fbf424b9e-kube-api-access-6n58s\") pod \"kuadrant-operator-controller-manager-55467d9f7-g6c54\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:36.022381 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.022350 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.092894 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.092860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8slg\" (UniqueName: \"kubernetes.io/projected/87e40348-9450-4eca-9438-8e04da9d7e80-kube-api-access-g8slg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x5wg4\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:36.093068 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.092951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87e40348-9450-4eca-9438-8e04da9d7e80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x5wg4\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:36.093068 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.093014 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshp9\" (UniqueName: \"kubernetes.io/projected/ce0708cc-7027-4f90-939c-0478520ae475-kube-api-access-pshp9\") pod \"limitador-operator-controller-manager-859d7b78bd-cfdpx\" (UID: \"ce0708cc-7027-4f90-939c-0478520ae475\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" Jul 09 14:20:36.093373 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.093345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87e40348-9450-4eca-9438-8e04da9d7e80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x5wg4\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:36.105254 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.105204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8slg\" (UniqueName: \"kubernetes.io/projected/87e40348-9450-4eca-9438-8e04da9d7e80-kube-api-access-g8slg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x5wg4\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:36.118983 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.118952 2575 generic.go:356] "Generic (PLEG): container finished" podID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" containerID="8d05c83c1d76c17c9b098a115dcd8a243b10c4fa5cb720f03d05b9390e708d49" exitCode=0 Jul 09 14:20:36.120907 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.120876 2575 generic.go:356] "Generic (PLEG): container finished" podID="cac733db-3d32-48e9-bb9f-19749da5bdbe" containerID="c6067df9211965243edbe78dad054098c2768a6c1fc2ba7f8ac6ec8ed905ba7a" exitCode=0 Jul 09 14:20:36.121024 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.120964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:36.121024 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.120989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:36.123250 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.123210 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.125364 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.125337 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.128885 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.128864 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:36.130058 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.130038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:36.132375 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.132352 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.134182 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.134159 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:36.134388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.134371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:36.139150 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.139127 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:36.139273 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.139193 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.141151 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.141126 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:36.142910 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.142884 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:36.143110 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.143090 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.144769 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.144743 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.146541 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.146517 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:36.148390 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.148367 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:36.150305 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.150278 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:36.152101 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.152079 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:36.153881 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:36.153862 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193373 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8rs7\" (UniqueName: \"kubernetes.io/projected/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-kube-api-access-z8rs7\") pod \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193429 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb7kl\" (UniqueName: \"kubernetes.io/projected/35820e70-2e70-4f9f-9a9e-12fe464ffe2b-kube-api-access-pb7kl\") pod \"35820e70-2e70-4f9f-9a9e-12fe464ffe2b\" (UID: \"35820e70-2e70-4f9f-9a9e-12fe464ffe2b\") " Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kklfn\" (UniqueName: \"kubernetes.io/projected/cac733db-3d32-48e9-bb9f-19749da5bdbe-kube-api-access-kklfn\") pod \"cac733db-3d32-48e9-bb9f-19749da5bdbe\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193558 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwm6r\" (UniqueName: \"kubernetes.io/projected/0ebf175c-e55f-446e-853a-f10400244c1d-kube-api-access-nwm6r\") pod \"0ebf175c-e55f-446e-853a-f10400244c1d\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193610 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac733db-3d32-48e9-bb9f-19749da5bdbe-extensions-socket-volume\") pod \"cac733db-3d32-48e9-bb9f-19749da5bdbe\" (UID: \"cac733db-3d32-48e9-bb9f-19749da5bdbe\") " Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193659 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-extensions-socket-volume\") pod \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\" (UID: \"1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2\") " Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193703 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ebf175c-e55f-446e-853a-f10400244c1d-extensions-socket-volume\") pod \"0ebf175c-e55f-446e-853a-f10400244c1d\" (UID: \"0ebf175c-e55f-446e-853a-f10400244c1d\") " Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.193944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshp9\" (UniqueName: \"kubernetes.io/projected/ce0708cc-7027-4f90-939c-0478520ae475-kube-api-access-pshp9\") pod \"limitador-operator-controller-manager-859d7b78bd-cfdpx\" (UID: \"ce0708cc-7027-4f90-939c-0478520ae475\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" Jul 09 14:20:36.194478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.194295 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" (UID: "1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:20:36.195064 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.194515 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebf175c-e55f-446e-853a-f10400244c1d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0ebf175c-e55f-446e-853a-f10400244c1d" (UID: "0ebf175c-e55f-446e-853a-f10400244c1d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:20:36.195064 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.194791 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac733db-3d32-48e9-bb9f-19749da5bdbe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cac733db-3d32-48e9-bb9f-19749da5bdbe" (UID: "cac733db-3d32-48e9-bb9f-19749da5bdbe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:20:36.195674 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.195647 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-kube-api-access-z8rs7" (OuterVolumeSpecName: "kube-api-access-z8rs7") pod "1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" (UID: "1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2"). InnerVolumeSpecName "kube-api-access-z8rs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:20:36.196350 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.196324 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebf175c-e55f-446e-853a-f10400244c1d-kube-api-access-nwm6r" (OuterVolumeSpecName: "kube-api-access-nwm6r") pod "0ebf175c-e55f-446e-853a-f10400244c1d" (UID: "0ebf175c-e55f-446e-853a-f10400244c1d"). InnerVolumeSpecName "kube-api-access-nwm6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:20:36.196808 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.196767 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac733db-3d32-48e9-bb9f-19749da5bdbe-kube-api-access-kklfn" (OuterVolumeSpecName: "kube-api-access-kklfn") pod "cac733db-3d32-48e9-bb9f-19749da5bdbe" (UID: "cac733db-3d32-48e9-bb9f-19749da5bdbe"). InnerVolumeSpecName "kube-api-access-kklfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:20:36.196963 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.196941 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35820e70-2e70-4f9f-9a9e-12fe464ffe2b-kube-api-access-pb7kl" (OuterVolumeSpecName: "kube-api-access-pb7kl") pod "35820e70-2e70-4f9f-9a9e-12fe464ffe2b" (UID: "35820e70-2e70-4f9f-9a9e-12fe464ffe2b"). InnerVolumeSpecName "kube-api-access-pb7kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:20:36.202466 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.202433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshp9\" (UniqueName: \"kubernetes.io/projected/ce0708cc-7027-4f90-939c-0478520ae475-kube-api-access-pshp9\") pod \"limitador-operator-controller-manager-859d7b78bd-cfdpx\" (UID: \"ce0708cc-7027-4f90-939c-0478520ae475\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" Jul 09 14:20:36.268765 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.268676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:36.281211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.281189 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" Jul 09 14:20:36.284730 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.284706 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54"] Jul 09 14:20:36.287472 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:20:36.287448 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6261c9b_3b32_42e7_bb73_984fbf424b9e.slice/crio-ad2bbfad7b573cac886c2b63abb0d4b7f5fc28e56d052243a3128265efa3a70e WatchSource:0}: Error finding container ad2bbfad7b573cac886c2b63abb0d4b7f5fc28e56d052243a3128265efa3a70e: Status 404 returned error can't find the container with id ad2bbfad7b573cac886c2b63abb0d4b7f5fc28e56d052243a3128265efa3a70e Jul 09 14:20:36.295160 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.295138 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwm6r\" (UniqueName: \"kubernetes.io/projected/0ebf175c-e55f-446e-853a-f10400244c1d-kube-api-access-nwm6r\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:36.295160 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.295161 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac733db-3d32-48e9-bb9f-19749da5bdbe-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:36.295376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.295171 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:36.295376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.295182 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ebf175c-e55f-446e-853a-f10400244c1d-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:36.295376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.295190 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8rs7\" (UniqueName: \"kubernetes.io/projected/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2-kube-api-access-z8rs7\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:36.295376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.295199 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7kl\" (UniqueName: \"kubernetes.io/projected/35820e70-2e70-4f9f-9a9e-12fe464ffe2b-kube-api-access-pb7kl\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:36.295376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.295208 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kklfn\" (UniqueName: \"kubernetes.io/projected/cac733db-3d32-48e9-bb9f-19749da5bdbe-kube-api-access-kklfn\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:36.438811 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:20:36.438771 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e40348_9450_4eca_9438_8e04da9d7e80.slice/crio-82a47162269fa3f5a77b51c0a3e04f1f9d51ad346fe476f328f09b3c0277c290 WatchSource:0}: Error finding container 82a47162269fa3f5a77b51c0a3e04f1f9d51ad346fe476f328f09b3c0277c290: Status 404 returned error can't find the container with id 82a47162269fa3f5a77b51c0a3e04f1f9d51ad346fe476f328f09b3c0277c290 Jul 09 14:20:36.438951 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.438920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4"] Jul 09 14:20:36.457554 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:36.457526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx"] Jul 09 14:20:36.459505 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:20:36.459472 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0708cc_7027_4f90_939c_0478520ae475.slice/crio-4acf5f39711e42823713b44cc5c9b6b050226025ac49d9ad91c921f93f3db7b5 WatchSource:0}: Error finding container 4acf5f39711e42823713b44cc5c9b6b050226025ac49d9ad91c921f93f3db7b5: Status 404 returned error can't find the container with id 4acf5f39711e42823713b44cc5c9b6b050226025ac49d9ad91c921f93f3db7b5 Jul 09 14:20:37.128297 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.128269 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8wqt9" Jul 09 14:20:37.128297 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.128284 2575 scope.go:117] "RemoveContainer" containerID="c6067df9211965243edbe78dad054098c2768a6c1fc2ba7f8ac6ec8ed905ba7a" Jul 09 14:20:37.129953 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.129922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" event={"ID":"e6261c9b-3b32-42e7-bb73-984fbf424b9e","Type":"ContainerStarted","Data":"26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e"} Jul 09 14:20:37.130087 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.129984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" event={"ID":"e6261c9b-3b32-42e7-bb73-984fbf424b9e","Type":"ContainerStarted","Data":"ad2bbfad7b573cac886c2b63abb0d4b7f5fc28e56d052243a3128265efa3a70e"} Jul 09 14:20:37.130087 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.130017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:37.130840 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.130806 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:37.131867 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.131836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" event={"ID":"87e40348-9450-4eca-9438-8e04da9d7e80","Type":"ContainerStarted","Data":"9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f"} Jul 09 14:20:37.131978 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.131893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" event={"ID":"87e40348-9450-4eca-9438-8e04da9d7e80","Type":"ContainerStarted","Data":"82a47162269fa3f5a77b51c0a3e04f1f9d51ad346fe476f328f09b3c0277c290"} Jul 09 14:20:37.131978 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.131951 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:37.133105 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.133072 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:37.133407 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.133385 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:37.135004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.134961 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" event={"ID":"ce0708cc-7027-4f90-939c-0478520ae475","Type":"ContainerStarted","Data":"46914ef55dbfd18f5a9cc474eca1907eae9a3b65f31e1c8f9a5d9ed96d9e1c38"} Jul 09 14:20:37.135114 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.135006 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:37.135114 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.135011 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" event={"ID":"ce0708cc-7027-4f90-939c-0478520ae475","Type":"ContainerStarted","Data":"4acf5f39711e42823713b44cc5c9b6b050226025ac49d9ad91c921f93f3db7b5"} Jul 09 14:20:37.135114 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.134977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:37.135114 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.135111 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" Jul 09 14:20:37.135347 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.135173 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:37.138683 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.138424 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:37.143641 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.142521 2575 scope.go:117] "RemoveContainer" containerID="8d05c83c1d76c17c9b098a115dcd8a243b10c4fa5cb720f03d05b9390e708d49" Jul 09 14:20:37.162025 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.161994 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:37.162617 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.162579 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" podStartSLOduration=2.162566696 podStartE2EDuration="2.162566696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:37.160093002 +0000 UTC m=+403.849022241" watchObservedRunningTime="2026-07-09 14:20:37.162566696 +0000 UTC m=+403.851495999" Jul 09 14:20:37.182827 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.182762 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" podStartSLOduration=2.1827448 podStartE2EDuration="2.1827448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:37.181052711 +0000 UTC m=+403.869981986" watchObservedRunningTime="2026-07-09 14:20:37.1827448 +0000 UTC m=+403.871674101" Jul 09 14:20:37.201626 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.201569 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" podStartSLOduration=2.201555055 podStartE2EDuration="2.201555055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:37.200430209 +0000 UTC m=+403.889359467" watchObservedRunningTime="2026-07-09 14:20:37.201555055 +0000 UTC m=+403.890484293" Jul 09 14:20:37.202441 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.202411 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:37.204404 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.204375 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6fzd5\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6fzd5" Jul 09 14:20:37.206161 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.206134 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6dsdq\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6dsdq" Jul 09 14:20:37.207926 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:37.207905 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-85cm8\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-85cm8" Jul 09 14:20:37.886685 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.886640 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ebf175c-e55f-446e-853a-f10400244c1d" path="/var/lib/kubelet/pods/0ebf175c-e55f-446e-853a-f10400244c1d/volumes" Jul 09 14:20:37.887171 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.887012 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2" path="/var/lib/kubelet/pods/1bb1425f-cdc5-42bd-9aa2-abc4ddcb49f2/volumes" Jul 09 14:20:37.887330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.887312 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35820e70-2e70-4f9f-9a9e-12fe464ffe2b" path="/var/lib/kubelet/pods/35820e70-2e70-4f9f-9a9e-12fe464ffe2b/volumes" Jul 09 14:20:37.887680 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:37.887662 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac733db-3d32-48e9-bb9f-19749da5bdbe" path="/var/lib/kubelet/pods/cac733db-3d32-48e9-bb9f-19749da5bdbe/volumes" Jul 09 14:20:48.143535 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.143503 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:48.144025 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.143580 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:20:48.144025 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.143749 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cfdpx" Jul 09 14:20:48.231255 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.231208 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4"] Jul 09 14:20:48.231494 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.231460 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" podUID="87e40348-9450-4eca-9438-8e04da9d7e80" containerName="manager" containerID="cri-o://9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f" gracePeriod=10 Jul 09 14:20:48.491770 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.491747 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:48.528148 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.528112 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg"] Jul 09 14:20:48.528557 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.528541 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e40348-9450-4eca-9438-8e04da9d7e80" containerName="manager" Jul 09 14:20:48.528557 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.528557 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e40348-9450-4eca-9438-8e04da9d7e80" containerName="manager" Jul 09 14:20:48.528682 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.528620 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e40348-9450-4eca-9438-8e04da9d7e80" containerName="manager" Jul 09 14:20:48.531310 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.531284 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.544282 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.544253 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg"] Jul 09 14:20:48.600300 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.600262 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8slg\" (UniqueName: \"kubernetes.io/projected/87e40348-9450-4eca-9438-8e04da9d7e80-kube-api-access-g8slg\") pod \"87e40348-9450-4eca-9438-8e04da9d7e80\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " Jul 09 14:20:48.600489 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.600315 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87e40348-9450-4eca-9438-8e04da9d7e80-extensions-socket-volume\") pod \"87e40348-9450-4eca-9438-8e04da9d7e80\" (UID: \"87e40348-9450-4eca-9438-8e04da9d7e80\") " Jul 09 14:20:48.600686 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.600659 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e40348-9450-4eca-9438-8e04da9d7e80-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "87e40348-9450-4eca-9438-8e04da9d7e80" (UID: "87e40348-9450-4eca-9438-8e04da9d7e80"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:20:48.602404 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.602379 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e40348-9450-4eca-9438-8e04da9d7e80-kube-api-access-g8slg" (OuterVolumeSpecName: "kube-api-access-g8slg") pod "87e40348-9450-4eca-9438-8e04da9d7e80" (UID: "87e40348-9450-4eca-9438-8e04da9d7e80"). InnerVolumeSpecName "kube-api-access-g8slg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:20:48.702067 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.701954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mcggg\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.702067 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.702020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrpf\" (UniqueName: \"kubernetes.io/projected/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-kube-api-access-zzrpf\") pod \"kuadrant-operator-controller-manager-86878c769f-mcggg\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.702317 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.702082 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8slg\" (UniqueName: \"kubernetes.io/projected/87e40348-9450-4eca-9438-8e04da9d7e80-kube-api-access-g8slg\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:48.702317 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.702097 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87e40348-9450-4eca-9438-8e04da9d7e80-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:20:48.802726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.802670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mcggg\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.802726 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.802728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrpf\" (UniqueName: \"kubernetes.io/projected/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-kube-api-access-zzrpf\") pod \"kuadrant-operator-controller-manager-86878c769f-mcggg\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.803099 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.803065 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mcggg\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.811945 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.811918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrpf\" (UniqueName: \"kubernetes.io/projected/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-kube-api-access-zzrpf\") pod \"kuadrant-operator-controller-manager-86878c769f-mcggg\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.842806 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.842762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:48.991400 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:48.991374 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg"] Jul 09 14:20:48.993407 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:20:48.993383 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b14c55_5a6f_41c2_8e1a_9c3060bcec94.slice/crio-6df7d703c8cb15f5cae8993e3c80cbc016b55b26ffe8cbffeea682d7340fe021 WatchSource:0}: Error finding container 6df7d703c8cb15f5cae8993e3c80cbc016b55b26ffe8cbffeea682d7340fe021: Status 404 returned error can't find the container with id 6df7d703c8cb15f5cae8993e3c80cbc016b55b26ffe8cbffeea682d7340fe021 Jul 09 14:20:49.196358 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.196319 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" event={"ID":"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94","Type":"ContainerStarted","Data":"706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a"} Jul 09 14:20:49.196817 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.196383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" event={"ID":"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94","Type":"ContainerStarted","Data":"6df7d703c8cb15f5cae8993e3c80cbc016b55b26ffe8cbffeea682d7340fe021"} Jul 09 14:20:49.196817 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.196518 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:20:49.197616 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.197593 2575 generic.go:356] "Generic (PLEG): container finished" podID="87e40348-9450-4eca-9438-8e04da9d7e80" containerID="9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f" exitCode=0 Jul 09 14:20:49.197729 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.197615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" event={"ID":"87e40348-9450-4eca-9438-8e04da9d7e80","Type":"ContainerDied","Data":"9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f"} Jul 09 14:20:49.197729 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.197636 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" Jul 09 14:20:49.197729 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.197655 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4" event={"ID":"87e40348-9450-4eca-9438-8e04da9d7e80","Type":"ContainerDied","Data":"82a47162269fa3f5a77b51c0a3e04f1f9d51ad346fe476f328f09b3c0277c290"} Jul 09 14:20:49.197729 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.197677 2575 scope.go:117] "RemoveContainer" containerID="9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f" Jul 09 14:20:49.206428 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.206413 2575 scope.go:117] "RemoveContainer" containerID="9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f" Jul 09 14:20:49.206680 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:20:49.206663 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f\": container with ID starting with 9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f not found: ID does not exist" containerID="9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f" Jul 09 14:20:49.206725 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.206691 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f"} err="failed to get container status \"9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f\": rpc error: code = NotFound desc = could not find container \"9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f\": container with ID starting with 9a5e92be942bb18b580cb4e0d8c7a1241f0d7bc5c004d6384cdb4b88030f027f not found: ID does not exist" Jul 09 14:20:49.221287 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.221205 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" podStartSLOduration=1.221191364 podStartE2EDuration="1.221191364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:49.220318679 +0000 UTC m=+415.909247943" watchObservedRunningTime="2026-07-09 14:20:49.221191364 +0000 UTC m=+415.910120604" Jul 09 14:20:49.238714 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.238685 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4"] Jul 09 14:20:49.243632 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.243550 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x5wg4"] Jul 09 14:20:49.887171 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:20:49.887135 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e40348-9450-4eca-9438-8e04da9d7e80" path="/var/lib/kubelet/pods/87e40348-9450-4eca-9438-8e04da9d7e80/volumes" Jul 09 14:21:00.207569 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.207537 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:21:00.260723 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.260691 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54"] Jul 09 14:21:00.260928 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.260905 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" podUID="e6261c9b-3b32-42e7-bb73-984fbf424b9e" containerName="manager" containerID="cri-o://26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e" gracePeriod=10 Jul 09 14:21:00.517816 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.517790 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:21:00.706304 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.706270 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n58s\" (UniqueName: \"kubernetes.io/projected/e6261c9b-3b32-42e7-bb73-984fbf424b9e-kube-api-access-6n58s\") pod \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " Jul 09 14:21:00.706496 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.706328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6261c9b-3b32-42e7-bb73-984fbf424b9e-extensions-socket-volume\") pod \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\" (UID: \"e6261c9b-3b32-42e7-bb73-984fbf424b9e\") " Jul 09 14:21:00.706760 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.706720 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6261c9b-3b32-42e7-bb73-984fbf424b9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e6261c9b-3b32-42e7-bb73-984fbf424b9e" (UID: "e6261c9b-3b32-42e7-bb73-984fbf424b9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:21:00.708522 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.708500 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6261c9b-3b32-42e7-bb73-984fbf424b9e-kube-api-access-6n58s" (OuterVolumeSpecName: "kube-api-access-6n58s") pod "e6261c9b-3b32-42e7-bb73-984fbf424b9e" (UID: "e6261c9b-3b32-42e7-bb73-984fbf424b9e"). InnerVolumeSpecName "kube-api-access-6n58s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:21:00.807100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.807055 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n58s\" (UniqueName: \"kubernetes.io/projected/e6261c9b-3b32-42e7-bb73-984fbf424b9e-kube-api-access-6n58s\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:21:00.807100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:00.807095 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6261c9b-3b32-42e7-bb73-984fbf424b9e-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:21:01.261656 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.261566 2575 generic.go:356] "Generic (PLEG): container finished" podID="e6261c9b-3b32-42e7-bb73-984fbf424b9e" containerID="26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e" exitCode=0 Jul 09 14:21:01.261656 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.261608 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" Jul 09 14:21:01.262121 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.261661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" event={"ID":"e6261c9b-3b32-42e7-bb73-984fbf424b9e","Type":"ContainerDied","Data":"26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e"} Jul 09 14:21:01.262121 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.261711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54" event={"ID":"e6261c9b-3b32-42e7-bb73-984fbf424b9e","Type":"ContainerDied","Data":"ad2bbfad7b573cac886c2b63abb0d4b7f5fc28e56d052243a3128265efa3a70e"} Jul 09 14:21:01.262121 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.261733 2575 scope.go:117] "RemoveContainer" containerID="26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e" Jul 09 14:21:01.271083 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.271063 2575 scope.go:117] "RemoveContainer" containerID="26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e" Jul 09 14:21:01.271369 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:21:01.271350 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e\": container with ID starting with 26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e not found: ID does not exist" containerID="26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e" Jul 09 14:21:01.271422 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.271378 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e"} err="failed to get container status \"26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e\": rpc error: code = NotFound desc = could not find container \"26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e\": container with ID starting with 26b1074c25346e16e836b30744782fdfa009f3bca0ddcb4e8d230b50fd63968e not found: ID does not exist" Jul 09 14:21:01.303365 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.303324 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54"] Jul 09 14:21:01.320364 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.320336 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-g6c54"] Jul 09 14:21:01.887172 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:01.887140 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6261c9b-3b32-42e7-bb73-984fbf424b9e" path="/var/lib/kubelet/pods/e6261c9b-3b32-42e7-bb73-984fbf424b9e/volumes" Jul 09 14:21:18.957739 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.957700 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhndf"] Jul 09 14:21:18.958158 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.958139 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6261c9b-3b32-42e7-bb73-984fbf424b9e" containerName="manager" Jul 09 14:21:18.958218 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.958161 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6261c9b-3b32-42e7-bb73-984fbf424b9e" containerName="manager" Jul 09 14:21:18.958289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.958277 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6261c9b-3b32-42e7-bb73-984fbf424b9e" containerName="manager" Jul 09 14:21:18.961451 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.961431 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:18.964162 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.964135 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 14:21:18.964294 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.964188 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gcv6t\"" Jul 09 14:21:18.974750 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:18.974720 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhndf"] Jul 09 14:21:19.054834 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.054785 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhndf"] Jul 09 14:21:19.059143 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.059113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgbg\" (UniqueName: \"kubernetes.io/projected/2d078879-9770-4c21-8f7d-6271e134afe7-kube-api-access-fzgbg\") pod \"limitador-limitador-64c94fd64f-mhndf\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:19.059327 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.059185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d078879-9770-4c21-8f7d-6271e134afe7-config-file\") pod \"limitador-limitador-64c94fd64f-mhndf\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:19.159782 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.159741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d078879-9770-4c21-8f7d-6271e134afe7-config-file\") pod \"limitador-limitador-64c94fd64f-mhndf\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:19.159998 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.159858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgbg\" (UniqueName: \"kubernetes.io/projected/2d078879-9770-4c21-8f7d-6271e134afe7-kube-api-access-fzgbg\") pod \"limitador-limitador-64c94fd64f-mhndf\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:19.160600 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.160571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d078879-9770-4c21-8f7d-6271e134afe7-config-file\") pod \"limitador-limitador-64c94fd64f-mhndf\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:19.171866 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.171829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgbg\" (UniqueName: \"kubernetes.io/projected/2d078879-9770-4c21-8f7d-6271e134afe7-kube-api-access-fzgbg\") pod \"limitador-limitador-64c94fd64f-mhndf\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:19.272071 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.272040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:19.422129 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:19.422105 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhndf"] Jul 09 14:21:19.424349 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:21:19.424303 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d078879_9770_4c21_8f7d_6271e134afe7.slice/crio-fde585628a41488ffd138b28b5811eb9c40f659d6e16d84a84299801e6478d5c WatchSource:0}: Error finding container fde585628a41488ffd138b28b5811eb9c40f659d6e16d84a84299801e6478d5c: Status 404 returned error can't find the container with id fde585628a41488ffd138b28b5811eb9c40f659d6e16d84a84299801e6478d5c Jul 09 14:21:20.356459 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:20.356423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" event={"ID":"2d078879-9770-4c21-8f7d-6271e134afe7","Type":"ContainerStarted","Data":"fde585628a41488ffd138b28b5811eb9c40f659d6e16d84a84299801e6478d5c"} Jul 09 14:21:24.379875 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:24.379835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" event={"ID":"2d078879-9770-4c21-8f7d-6271e134afe7","Type":"ContainerStarted","Data":"cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056"} Jul 09 14:21:24.380334 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:24.379988 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:24.400402 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:24.400355 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" podStartSLOduration=2.369940979 podStartE2EDuration="6.400339265s" totalImagesPullingTime="4.030398286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:24.399428743 +0000 UTC m=+451.088358004" watchObservedRunningTime="2026-07-09 14:21:24.400339265 +0000 UTC m=+451.089268505" Jul 09 14:21:35.067280 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.067240 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhndf"] Jul 09 14:21:35.067672 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.067432 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" podUID="2d078879-9770-4c21-8f7d-6271e134afe7" containerName="limitador" containerID="cri-o://cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056" gracePeriod=30 Jul 09 14:21:35.068207 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.068177 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:35.625564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.625539 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:35.711566 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.711470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzgbg\" (UniqueName: \"kubernetes.io/projected/2d078879-9770-4c21-8f7d-6271e134afe7-kube-api-access-fzgbg\") pod \"2d078879-9770-4c21-8f7d-6271e134afe7\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " Jul 09 14:21:35.711566 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.711560 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d078879-9770-4c21-8f7d-6271e134afe7-config-file\") pod \"2d078879-9770-4c21-8f7d-6271e134afe7\" (UID: \"2d078879-9770-4c21-8f7d-6271e134afe7\") " Jul 09 14:21:35.711932 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.711907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d078879-9770-4c21-8f7d-6271e134afe7-config-file" (OuterVolumeSpecName: "config-file") pod "2d078879-9770-4c21-8f7d-6271e134afe7" (UID: "2d078879-9770-4c21-8f7d-6271e134afe7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:21:35.713632 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.713602 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d078879-9770-4c21-8f7d-6271e134afe7-kube-api-access-fzgbg" (OuterVolumeSpecName: "kube-api-access-fzgbg") pod "2d078879-9770-4c21-8f7d-6271e134afe7" (UID: "2d078879-9770-4c21-8f7d-6271e134afe7"). InnerVolumeSpecName "kube-api-access-fzgbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:21:35.812933 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.812896 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgbg\" (UniqueName: \"kubernetes.io/projected/2d078879-9770-4c21-8f7d-6271e134afe7-kube-api-access-fzgbg\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:21:35.812933 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:35.812927 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d078879-9770-4c21-8f7d-6271e134afe7-config-file\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:21:36.445464 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.445427 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d078879-9770-4c21-8f7d-6271e134afe7" containerID="cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056" exitCode=0 Jul 09 14:21:36.445464 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.445463 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" Jul 09 14:21:36.445995 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.445514 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" event={"ID":"2d078879-9770-4c21-8f7d-6271e134afe7","Type":"ContainerDied","Data":"cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056"} Jul 09 14:21:36.445995 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.445565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhndf" event={"ID":"2d078879-9770-4c21-8f7d-6271e134afe7","Type":"ContainerDied","Data":"fde585628a41488ffd138b28b5811eb9c40f659d6e16d84a84299801e6478d5c"} Jul 09 14:21:36.445995 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.445588 2575 scope.go:117] "RemoveContainer" containerID="cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056" Jul 09 14:21:36.454442 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.454422 2575 scope.go:117] "RemoveContainer" containerID="cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056" Jul 09 14:21:36.454728 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:21:36.454707 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056\": container with ID starting with cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056 not found: ID does not exist" containerID="cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056" Jul 09 14:21:36.454790 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.454743 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056"} err="failed to get container status \"cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056\": rpc error: code = NotFound desc = could not find container \"cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056\": container with ID starting with cf56ec9738223cfef4235b80c9836d65ac4677a055bf0822b35dcd0122ec5056 not found: ID does not exist" Jul 09 14:21:36.462459 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.462434 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhndf"] Jul 09 14:21:36.467432 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:36.467409 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhndf"] Jul 09 14:21:37.886023 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:37.885989 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d078879-9770-4c21-8f7d-6271e134afe7" path="/var/lib/kubelet/pods/2d078879-9770-4c21-8f7d-6271e134afe7/volumes" Jul 09 14:21:39.087201 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.087167 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p"] Jul 09 14:21:39.087687 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.087551 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d078879-9770-4c21-8f7d-6271e134afe7" containerName="limitador" Jul 09 14:21:39.087687 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.087564 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d078879-9770-4c21-8f7d-6271e134afe7" containerName="limitador" Jul 09 14:21:39.087687 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.087651 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d078879-9770-4c21-8f7d-6271e134afe7" containerName="limitador" Jul 09 14:21:39.094366 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.093551 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.098336 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.098306 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d7c6k\"" Jul 09 14:21:39.098493 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.098306 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 14:21:39.098493 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.098310 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 14:21:39.099409 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.099237 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 14:21:39.100027 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.100005 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p"] Jul 09 14:21:39.146470 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.146429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eba1615b-ea0b-4afc-8b15-98b8eb48ada5-data\") pod \"postgres-69fb4594cf-9tc6p\" (UID: \"eba1615b-ea0b-4afc-8b15-98b8eb48ada5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.146623 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.146480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxk9\" (UniqueName: \"kubernetes.io/projected/eba1615b-ea0b-4afc-8b15-98b8eb48ada5-kube-api-access-8lxk9\") pod \"postgres-69fb4594cf-9tc6p\" (UID: \"eba1615b-ea0b-4afc-8b15-98b8eb48ada5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.247416 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.247375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eba1615b-ea0b-4afc-8b15-98b8eb48ada5-data\") pod \"postgres-69fb4594cf-9tc6p\" (UID: \"eba1615b-ea0b-4afc-8b15-98b8eb48ada5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.247607 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.247431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxk9\" (UniqueName: \"kubernetes.io/projected/eba1615b-ea0b-4afc-8b15-98b8eb48ada5-kube-api-access-8lxk9\") pod \"postgres-69fb4594cf-9tc6p\" (UID: \"eba1615b-ea0b-4afc-8b15-98b8eb48ada5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.247749 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.247724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eba1615b-ea0b-4afc-8b15-98b8eb48ada5-data\") pod \"postgres-69fb4594cf-9tc6p\" (UID: \"eba1615b-ea0b-4afc-8b15-98b8eb48ada5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.255947 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.255918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxk9\" (UniqueName: \"kubernetes.io/projected/eba1615b-ea0b-4afc-8b15-98b8eb48ada5-kube-api-access-8lxk9\") pod \"postgres-69fb4594cf-9tc6p\" (UID: \"eba1615b-ea0b-4afc-8b15-98b8eb48ada5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.406073 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.405979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:39.541068 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:39.541044 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p"] Jul 09 14:21:39.543160 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:21:39.543131 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba1615b_ea0b_4afc_8b15_98b8eb48ada5.slice/crio-cf61b08c18ed7174ccd7108c658ec9550c10b4fcc25c98ad60005303fa663ef9 WatchSource:0}: Error finding container cf61b08c18ed7174ccd7108c658ec9550c10b4fcc25c98ad60005303fa663ef9: Status 404 returned error can't find the container with id cf61b08c18ed7174ccd7108c658ec9550c10b4fcc25c98ad60005303fa663ef9 Jul 09 14:21:40.468558 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:40.468520 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" event={"ID":"eba1615b-ea0b-4afc-8b15-98b8eb48ada5","Type":"ContainerStarted","Data":"cf61b08c18ed7174ccd7108c658ec9550c10b4fcc25c98ad60005303fa663ef9"} Jul 09 14:21:45.496833 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:45.496796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" event={"ID":"eba1615b-ea0b-4afc-8b15-98b8eb48ada5","Type":"ContainerStarted","Data":"6ad762aa113b02a6f239434c9f28c4d8821368967aa070600072ecb4abde64c3"} Jul 09 14:21:45.497204 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:45.496973 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:45.514453 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:45.514392 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" podStartSLOduration=1.470481969 podStartE2EDuration="6.514378658s" totalImagesPullingTime="5.043896689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:45.511928873 +0000 UTC m=+472.200858112" watchObservedRunningTime="2026-07-09 14:21:45.514378658 +0000 UTC m=+472.203307897" Jul 09 14:21:51.531973 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:51.531945 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9tc6p" Jul 09 14:21:52.657444 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.657384 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bfd645988-cpclm"] Jul 09 14:21:52.661048 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.661031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:52.664846 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.664820 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wtqm6\"" Jul 09 14:21:52.664964 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.664821 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 14:21:52.669691 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.669661 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bfd645988-cpclm"] Jul 09 14:21:52.774772 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.774717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac03e1b2-ad12-49d9-8340-1747db4b44a4-tls-cert\") pod \"authorino-bfd645988-cpclm\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:52.774772 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.774774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7f54\" (UniqueName: \"kubernetes.io/projected/ac03e1b2-ad12-49d9-8340-1747db4b44a4-kube-api-access-s7f54\") pod \"authorino-bfd645988-cpclm\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:52.875545 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.875511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7f54\" (UniqueName: \"kubernetes.io/projected/ac03e1b2-ad12-49d9-8340-1747db4b44a4-kube-api-access-s7f54\") pod \"authorino-bfd645988-cpclm\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:52.875714 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.875611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac03e1b2-ad12-49d9-8340-1747db4b44a4-tls-cert\") pod \"authorino-bfd645988-cpclm\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:52.877933 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.877915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac03e1b2-ad12-49d9-8340-1747db4b44a4-tls-cert\") pod \"authorino-bfd645988-cpclm\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:52.889368 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.889338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7f54\" (UniqueName: \"kubernetes.io/projected/ac03e1b2-ad12-49d9-8340-1747db4b44a4-kube-api-access-s7f54\") pod \"authorino-bfd645988-cpclm\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:52.971417 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:52.971333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:21:53.110569 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:53.110543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bfd645988-cpclm"] Jul 09 14:21:53.112236 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:21:53.112192 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac03e1b2_ad12_49d9_8340_1747db4b44a4.slice/crio-625841d71acec8a42ad09802e7efa4bfbff8524d14f8ad09b5f479330d0e22dd WatchSource:0}: Error finding container 625841d71acec8a42ad09802e7efa4bfbff8524d14f8ad09b5f479330d0e22dd: Status 404 returned error can't find the container with id 625841d71acec8a42ad09802e7efa4bfbff8524d14f8ad09b5f479330d0e22dd Jul 09 14:21:53.537770 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:53.537735 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bfd645988-cpclm" event={"ID":"ac03e1b2-ad12-49d9-8340-1747db4b44a4","Type":"ContainerStarted","Data":"625841d71acec8a42ad09802e7efa4bfbff8524d14f8ad09b5f479330d0e22dd"} Jul 09 14:21:56.557566 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:56.557527 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bfd645988-cpclm" event={"ID":"ac03e1b2-ad12-49d9-8340-1747db4b44a4","Type":"ContainerStarted","Data":"89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a"} Jul 09 14:21:56.574069 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:21:56.574014 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bfd645988-cpclm" podStartSLOduration=1.262289491 podStartE2EDuration="4.573997469s" totalImagesPullingTime="3.311707978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:56.573912897 +0000 UTC m=+483.262842136" watchObservedRunningTime="2026-07-09 14:21:56.573997469 +0000 UTC m=+483.262926708" Jul 09 14:23:01.193831 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.193745 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl"] Jul 09 14:23:01.197427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.197408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.200046 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.200026 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 14:23:01.200154 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.200113 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 14:23:01.201261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.201195 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rmz94\"" Jul 09 14:23:01.201261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.201258 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 14:23:01.207385 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.207361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl"] Jul 09 14:23:01.270214 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.270180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.270401 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.270216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3a77b9-7878-49bf-bcb2-e2f429361bef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.270401 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.270316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.270485 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.270407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbfh\" (UniqueName: \"kubernetes.io/projected/fc3a77b9-7878-49bf-bcb2-e2f429361bef-kube-api-access-8sbfh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.270485 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.270444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.270485 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.270460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.371603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.371566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.371603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.371602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.371844 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.371627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.371844 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.371651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3a77b9-7878-49bf-bcb2-e2f429361bef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.371844 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.371705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.371844 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.371787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbfh\" (UniqueName: \"kubernetes.io/projected/fc3a77b9-7878-49bf-bcb2-e2f429361bef-kube-api-access-8sbfh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.372063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.372039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.372063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.372056 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.372173 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.372155 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.373935 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.373912 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3a77b9-7878-49bf-bcb2-e2f429361bef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.374185 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.374168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3a77b9-7878-49bf-bcb2-e2f429361bef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.380021 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.379999 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbfh\" (UniqueName: \"kubernetes.io/projected/fc3a77b9-7878-49bf-bcb2-e2f429361bef-kube-api-access-8sbfh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl\" (UID: \"fc3a77b9-7878-49bf-bcb2-e2f429361bef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.509165 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.509070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:01.651655 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.651622 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl"] Jul 09 14:23:01.654136 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:23:01.654107 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3a77b9_7878_49bf_bcb2_e2f429361bef.slice/crio-1d9e195ac1e036226033d9733e1605cfc4567c531caa122b81a70c9484e1cd1a WatchSource:0}: Error finding container 1d9e195ac1e036226033d9733e1605cfc4567c531caa122b81a70c9484e1cd1a: Status 404 returned error can't find the container with id 1d9e195ac1e036226033d9733e1605cfc4567c531caa122b81a70c9484e1cd1a Jul 09 14:23:01.886272 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:01.886205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" event={"ID":"fc3a77b9-7878-49bf-bcb2-e2f429361bef","Type":"ContainerStarted","Data":"1d9e195ac1e036226033d9733e1605cfc4567c531caa122b81a70c9484e1cd1a"} Jul 09 14:23:03.891052 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:03.891010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" event={"ID":"fc3a77b9-7878-49bf-bcb2-e2f429361bef","Type":"ContainerStarted","Data":"b1f7bbc3e13b20e97279f2673802d01786fec587b2985bc04185e01b17580071"} Jul 09 14:23:03.911739 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:03.911688 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" podStartSLOduration=1.6311157139999999 podStartE2EDuration="2.911671568s" totalImagesPullingTime="1.280555854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:23:03.910572933 +0000 UTC m=+550.599502176" watchObservedRunningTime="2026-07-09 14:23:03.911671568 +0000 UTC m=+550.600600807" Jul 09 14:23:11.510058 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:11.510020 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:11.510520 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:11.510134 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:11.522775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:11.522754 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:11.940460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:11.940421 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl" Jul 09 14:23:18.160353 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.160321 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk"] Jul 09 14:23:18.163863 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.163840 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.166421 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.166398 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 09 14:23:18.176149 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.176124 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk"] Jul 09 14:23:18.220089 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.220049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.220089 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.220098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.220388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.220168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.220388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.220206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e863a573-9ead-45ad-a14b-7af95a8f10f7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.220388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.220292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.220388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.220350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qwr\" (UniqueName: \"kubernetes.io/projected/e863a573-9ead-45ad-a14b-7af95a8f10f7-kube-api-access-p2qwr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.321720 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.321687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.321882 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.321838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qwr\" (UniqueName: \"kubernetes.io/projected/e863a573-9ead-45ad-a14b-7af95a8f10f7-kube-api-access-p2qwr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.321932 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.321898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.321980 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.321941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.322037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.321981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.322037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.322004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e863a573-9ead-45ad-a14b-7af95a8f10f7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.322351 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.322327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.322460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.322364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.322460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.322407 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.323831 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.323804 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e863a573-9ead-45ad-a14b-7af95a8f10f7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.324441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.324423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e863a573-9ead-45ad-a14b-7af95a8f10f7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.330605 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.330582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qwr\" (UniqueName: \"kubernetes.io/projected/e863a573-9ead-45ad-a14b-7af95a8f10f7-kube-api-access-p2qwr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nhtwk\" (UID: \"e863a573-9ead-45ad-a14b-7af95a8f10f7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.474740 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.474645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:18.621055 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.621030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk"] Jul 09 14:23:18.623318 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:23:18.623292 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode863a573_9ead_45ad_a14b_7af95a8f10f7.slice/crio-0fd549cbfc8aaf57b0aef38e4a8696cebfdc02ba95e0326a0dd1c4d30a3399cb WatchSource:0}: Error finding container 0fd549cbfc8aaf57b0aef38e4a8696cebfdc02ba95e0326a0dd1c4d30a3399cb: Status 404 returned error can't find the container with id 0fd549cbfc8aaf57b0aef38e4a8696cebfdc02ba95e0326a0dd1c4d30a3399cb Jul 09 14:23:18.965935 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.965894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" event={"ID":"e863a573-9ead-45ad-a14b-7af95a8f10f7","Type":"ContainerStarted","Data":"2c4d1b1d868ddb5d7ce4c35a46c9e75a187110ff18a5d6a25482a36ef95d0886"} Jul 09 14:23:18.966079 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.965952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" event={"ID":"e863a573-9ead-45ad-a14b-7af95a8f10f7","Type":"ContainerStarted","Data":"0fd549cbfc8aaf57b0aef38e4a8696cebfdc02ba95e0326a0dd1c4d30a3399cb"} Jul 09 14:23:18.986952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:18.986892 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" podStartSLOduration=0.742284378 podStartE2EDuration="986.874845ms" totalImagesPullingTime="244.590467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:23:18.985918309 +0000 UTC m=+565.674847569" watchObservedRunningTime="2026-07-09 14:23:18.986874845 +0000 UTC m=+565.675804084" Jul 09 14:23:28.474967 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:28.474928 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:28.475446 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:28.475000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:28.487143 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:28.487122 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:29.029683 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:29.029655 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nhtwk" Jul 09 14:23:53.786347 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:53.786314 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:23:53.788326 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:23:53.788306 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:24:20.714009 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:20.713963 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bfd645988-cpclm"] Jul 09 14:24:20.716573 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:20.714148 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bfd645988-cpclm" podUID="ac03e1b2-ad12-49d9-8340-1747db4b44a4" containerName="authorino" containerID="cri-o://89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a" gracePeriod=30 Jul 09 14:24:20.967597 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:20.967530 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:24:21.100564 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.100526 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7f54\" (UniqueName: \"kubernetes.io/projected/ac03e1b2-ad12-49d9-8340-1747db4b44a4-kube-api-access-s7f54\") pod \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " Jul 09 14:24:21.100767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.100634 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac03e1b2-ad12-49d9-8340-1747db4b44a4-tls-cert\") pod \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\" (UID: \"ac03e1b2-ad12-49d9-8340-1747db4b44a4\") " Jul 09 14:24:21.102733 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.102703 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac03e1b2-ad12-49d9-8340-1747db4b44a4-kube-api-access-s7f54" (OuterVolumeSpecName: "kube-api-access-s7f54") pod "ac03e1b2-ad12-49d9-8340-1747db4b44a4" (UID: "ac03e1b2-ad12-49d9-8340-1747db4b44a4"). InnerVolumeSpecName "kube-api-access-s7f54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:24:21.111879 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.111848 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac03e1b2-ad12-49d9-8340-1747db4b44a4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ac03e1b2-ad12-49d9-8340-1747db4b44a4" (UID: "ac03e1b2-ad12-49d9-8340-1747db4b44a4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:24:21.201780 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.201739 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac03e1b2-ad12-49d9-8340-1747db4b44a4-tls-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:24:21.201780 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.201771 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7f54\" (UniqueName: \"kubernetes.io/projected/ac03e1b2-ad12-49d9-8340-1747db4b44a4-kube-api-access-s7f54\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:24:21.286147 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.286110 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac03e1b2-ad12-49d9-8340-1747db4b44a4" containerID="89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a" exitCode=0 Jul 09 14:24:21.286330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.286153 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bfd645988-cpclm" Jul 09 14:24:21.286330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.286155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bfd645988-cpclm" event={"ID":"ac03e1b2-ad12-49d9-8340-1747db4b44a4","Type":"ContainerDied","Data":"89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a"} Jul 09 14:24:21.286330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.286195 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bfd645988-cpclm" event={"ID":"ac03e1b2-ad12-49d9-8340-1747db4b44a4","Type":"ContainerDied","Data":"625841d71acec8a42ad09802e7efa4bfbff8524d14f8ad09b5f479330d0e22dd"} Jul 09 14:24:21.286330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.286218 2575 scope.go:117] "RemoveContainer" containerID="89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a" Jul 09 14:24:21.295782 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.295758 2575 scope.go:117] "RemoveContainer" containerID="89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a" Jul 09 14:24:21.296096 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:24:21.296059 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a\": container with ID starting with 89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a not found: ID does not exist" containerID="89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a" Jul 09 14:24:21.296149 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.296104 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a"} err="failed to get container status \"89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a\": rpc error: code = NotFound desc = could not find container \"89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a\": container with ID starting with 89b407db01d25bebe2237d6b7ebbe94bf257ebbbd378a6aaadfe2398d467c09a not found: ID does not exist" Jul 09 14:24:21.325947 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.325915 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bfd645988-cpclm"] Jul 09 14:24:21.331029 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.330995 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bfd645988-cpclm"] Jul 09 14:24:21.886976 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:24:21.886947 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac03e1b2-ad12-49d9-8340-1747db4b44a4" path="/var/lib/kubelet/pods/ac03e1b2-ad12-49d9-8340-1747db4b44a4/volumes" Jul 09 14:28:53.819143 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:28:53.819057 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:28:53.823779 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:28:53.823753 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:30:00.152916 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.152881 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l"] Jul 09 14:30:00.155379 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.153283 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac03e1b2-ad12-49d9-8340-1747db4b44a4" containerName="authorino" Jul 09 14:30:00.155379 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.153296 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac03e1b2-ad12-49d9-8340-1747db4b44a4" containerName="authorino" Jul 09 14:30:00.155379 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.153362 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac03e1b2-ad12-49d9-8340-1747db4b44a4" containerName="authorino" Jul 09 14:30:00.156418 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.156402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" Jul 09 14:30:00.159391 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.159370 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bmdnz\"" Jul 09 14:30:00.174908 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.174876 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l"] Jul 09 14:30:00.189952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.189919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfzb\" (UniqueName: \"kubernetes.io/projected/895ee156-af10-456e-a11a-dc8e34910e9f-kube-api-access-dcfzb\") pod \"maas-api-key-cleanup-29726790-67j9l\" (UID: \"895ee156-af10-456e-a11a-dc8e34910e9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" Jul 09 14:30:00.290819 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.290782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfzb\" (UniqueName: \"kubernetes.io/projected/895ee156-af10-456e-a11a-dc8e34910e9f-kube-api-access-dcfzb\") pod \"maas-api-key-cleanup-29726790-67j9l\" (UID: \"895ee156-af10-456e-a11a-dc8e34910e9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" Jul 09 14:30:00.300526 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.300494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfzb\" (UniqueName: \"kubernetes.io/projected/895ee156-af10-456e-a11a-dc8e34910e9f-kube-api-access-dcfzb\") pod \"maas-api-key-cleanup-29726790-67j9l\" (UID: \"895ee156-af10-456e-a11a-dc8e34910e9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" Jul 09 14:30:00.466689 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.466604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" Jul 09 14:30:00.601493 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.601466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l"] Jul 09 14:30:00.603577 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:30:00.603537 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895ee156_af10_456e_a11a_dc8e34910e9f.slice/crio-e0fdd73e93e8aa74ce44bfd8c7c3d6133a0a39745448733eb38d60d08d2792f8 WatchSource:0}: Error finding container e0fdd73e93e8aa74ce44bfd8c7c3d6133a0a39745448733eb38d60d08d2792f8: Status 404 returned error can't find the container with id e0fdd73e93e8aa74ce44bfd8c7c3d6133a0a39745448733eb38d60d08d2792f8 Jul 09 14:30:00.605516 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.605496 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:30:00.988127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:00.988096 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" event={"ID":"895ee156-af10-456e-a11a-dc8e34910e9f","Type":"ContainerStarted","Data":"e0fdd73e93e8aa74ce44bfd8c7c3d6133a0a39745448733eb38d60d08d2792f8"} Jul 09 14:30:04.005556 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:04.005519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" event={"ID":"895ee156-af10-456e-a11a-dc8e34910e9f","Type":"ContainerStarted","Data":"eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527"} Jul 09 14:30:04.024504 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:04.024451 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" podStartSLOduration=1.710336442 podStartE2EDuration="4.024435741s" totalImagesPullingTime="2.314099299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:30:04.023125005 +0000 UTC m=+970.712054243" watchObservedRunningTime="2026-07-09 14:30:04.024435741 +0000 UTC m=+970.713364981" Jul 09 14:30:49.041909 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:49.041874 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg"] Jul 09 14:30:49.042444 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:49.042077 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" podUID="b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" containerName="manager" containerID="cri-o://706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a" gracePeriod=10 Jul 09 14:30:52.397854 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:52.397818 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:30:52.466530 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:52.466495 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzrpf\" (UniqueName: \"kubernetes.io/projected/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-kube-api-access-zzrpf\") pod \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " Jul 09 14:30:52.466530 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:52.466533 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-extensions-socket-volume\") pod \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\" (UID: \"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94\") " Jul 09 14:30:52.466988 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:52.466955 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" (UID: "b2b14c55-5a6f-41c2-8e1a-9c3060bcec94"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:30:52.468711 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:52.468681 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-kube-api-access-zzrpf" (OuterVolumeSpecName: "kube-api-access-zzrpf") pod "b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" (UID: "b2b14c55-5a6f-41c2-8e1a-9c3060bcec94"). InnerVolumeSpecName "kube-api-access-zzrpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:30:52.567765 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:52.567729 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrpf\" (UniqueName: \"kubernetes.io/projected/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-kube-api-access-zzrpf\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:30:52.567765 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:52.567758 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94-extensions-socket-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:30:53.255709 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.255672 2575 generic.go:356] "Generic (PLEG): container finished" podID="b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" containerID="706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a" exitCode=0 Jul 09 14:30:53.255898 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.255720 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" Jul 09 14:30:53.255898 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.255752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" event={"ID":"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94","Type":"ContainerDied","Data":"706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a"} Jul 09 14:30:53.255898 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.255809 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg" event={"ID":"b2b14c55-5a6f-41c2-8e1a-9c3060bcec94","Type":"ContainerDied","Data":"6df7d703c8cb15f5cae8993e3c80cbc016b55b26ffe8cbffeea682d7340fe021"} Jul 09 14:30:53.255898 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.255832 2575 scope.go:117] "RemoveContainer" containerID="706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a" Jul 09 14:30:53.266875 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.266859 2575 scope.go:117] "RemoveContainer" containerID="706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a" Jul 09 14:30:53.267177 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:30:53.267158 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a\": container with ID starting with 706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a not found: ID does not exist" containerID="706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a" Jul 09 14:30:53.267237 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.267186 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a"} err="failed to get container status \"706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a\": rpc error: code = NotFound desc = could not find container \"706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a\": container with ID starting with 706b91297f64e5a926f00a9e1db3d44cae1f1603a143babc7ae454425023d36a not found: ID does not exist" Jul 09 14:30:53.279654 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.279630 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg"] Jul 09 14:30:53.284237 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.284197 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mcggg"] Jul 09 14:30:53.886043 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:30:53.886009 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" path="/var/lib/kubelet/pods/b2b14c55-5a6f-41c2-8e1a-9c3060bcec94/volumes" Jul 09 14:31:00.011674 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:00.011623 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l"] Jul 09 14:31:00.012263 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:00.011836 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" podUID="895ee156-af10-456e-a11a-dc8e34910e9f" containerName="cleanup" containerID="cri-o://eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527" gracePeriod=30 Jul 09 14:31:30.204463 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.204436 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" Jul 09 14:31:30.292632 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.292598 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcfzb\" (UniqueName: \"kubernetes.io/projected/895ee156-af10-456e-a11a-dc8e34910e9f-kube-api-access-dcfzb\") pod \"895ee156-af10-456e-a11a-dc8e34910e9f\" (UID: \"895ee156-af10-456e-a11a-dc8e34910e9f\") " Jul 09 14:31:30.301083 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.301036 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895ee156-af10-456e-a11a-dc8e34910e9f-kube-api-access-dcfzb" (OuterVolumeSpecName: "kube-api-access-dcfzb") pod "895ee156-af10-456e-a11a-dc8e34910e9f" (UID: "895ee156-af10-456e-a11a-dc8e34910e9f"). InnerVolumeSpecName "kube-api-access-dcfzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:31:30.394151 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.394112 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfzb\" (UniqueName: \"kubernetes.io/projected/895ee156-af10-456e-a11a-dc8e34910e9f-kube-api-access-dcfzb\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:31:30.450154 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.450118 2575 generic.go:356] "Generic (PLEG): container finished" podID="895ee156-af10-456e-a11a-dc8e34910e9f" containerID="eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527" exitCode=137 Jul 09 14:31:30.450361 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.450179 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" Jul 09 14:31:30.450361 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.450185 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" event={"ID":"895ee156-af10-456e-a11a-dc8e34910e9f","Type":"ContainerDied","Data":"eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527"} Jul 09 14:31:30.450361 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.450272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l" event={"ID":"895ee156-af10-456e-a11a-dc8e34910e9f","Type":"ContainerDied","Data":"e0fdd73e93e8aa74ce44bfd8c7c3d6133a0a39745448733eb38d60d08d2792f8"} Jul 09 14:31:30.450361 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.450294 2575 scope.go:117] "RemoveContainer" containerID="eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527" Jul 09 14:31:30.459134 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.459118 2575 scope.go:117] "RemoveContainer" containerID="eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527" Jul 09 14:31:30.459429 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:31:30.459412 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527\": container with ID starting with eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527 not found: ID does not exist" containerID="eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527" Jul 09 14:31:30.459484 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.459437 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527"} err="failed to get container status \"eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527\": rpc error: code = NotFound desc = could not find container \"eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527\": container with ID starting with eb662fe5156ab251342a406cd8a90a2c78d1f801f8df21278689fc28114a1527 not found: ID does not exist" Jul 09 14:31:30.473946 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.473921 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l"] Jul 09 14:31:30.477644 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:30.477619 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726790-67j9l"] Jul 09 14:31:31.887717 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:31.887682 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895ee156-af10-456e-a11a-dc8e34910e9f" path="/var/lib/kubelet/pods/895ee156-af10-456e-a11a-dc8e34910e9f/volumes" Jul 09 14:31:50.180119 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.180036 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45"] Jul 09 14:31:50.180612 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.180464 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895ee156-af10-456e-a11a-dc8e34910e9f" containerName="cleanup" Jul 09 14:31:50.180612 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.180476 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="895ee156-af10-456e-a11a-dc8e34910e9f" containerName="cleanup" Jul 09 14:31:50.180612 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.180505 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" containerName="manager" Jul 09 14:31:50.180612 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.180511 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" containerName="manager" Jul 09 14:31:50.180612 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.180577 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b14c55-5a6f-41c2-8e1a-9c3060bcec94" containerName="manager" Jul 09 14:31:50.180612 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.180588 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="895ee156-af10-456e-a11a-dc8e34910e9f" containerName="cleanup" Jul 09 14:31:50.183627 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.183609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.188492 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.188471 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-csmzx\"" Jul 09 14:31:50.202124 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.202101 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45"] Jul 09 14:31:50.262100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.262063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8509b3b-9f7c-4dfc-a688-822903dd79ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cmn45\" (UID: \"c8509b3b-9f7c-4dfc-a688-822903dd79ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.262332 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.262200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8x2f\" (UniqueName: \"kubernetes.io/projected/c8509b3b-9f7c-4dfc-a688-822903dd79ba-kube-api-access-s8x2f\") pod \"kuadrant-operator-controller-manager-86878c769f-cmn45\" (UID: \"c8509b3b-9f7c-4dfc-a688-822903dd79ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.363406 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.363367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8x2f\" (UniqueName: \"kubernetes.io/projected/c8509b3b-9f7c-4dfc-a688-822903dd79ba-kube-api-access-s8x2f\") pod \"kuadrant-operator-controller-manager-86878c769f-cmn45\" (UID: \"c8509b3b-9f7c-4dfc-a688-822903dd79ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.363406 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.363412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8509b3b-9f7c-4dfc-a688-822903dd79ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cmn45\" (UID: \"c8509b3b-9f7c-4dfc-a688-822903dd79ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.363750 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.363733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8509b3b-9f7c-4dfc-a688-822903dd79ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cmn45\" (UID: \"c8509b3b-9f7c-4dfc-a688-822903dd79ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.377342 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.377311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8x2f\" (UniqueName: \"kubernetes.io/projected/c8509b3b-9f7c-4dfc-a688-822903dd79ba-kube-api-access-s8x2f\") pod \"kuadrant-operator-controller-manager-86878c769f-cmn45\" (UID: \"c8509b3b-9f7c-4dfc-a688-822903dd79ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.493889 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.493790 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:50.634891 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:50.634868 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45"] Jul 09 14:31:50.637306 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:31:50.637275 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8509b3b_9f7c_4dfc_a688_822903dd79ba.slice/crio-f408c9e6fbf40a8207fcd948a55fcf137e9682c835beb82995592de1b405cb5b WatchSource:0}: Error finding container f408c9e6fbf40a8207fcd948a55fcf137e9682c835beb82995592de1b405cb5b: Status 404 returned error can't find the container with id f408c9e6fbf40a8207fcd948a55fcf137e9682c835beb82995592de1b405cb5b Jul 09 14:31:51.558123 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:51.558088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" event={"ID":"c8509b3b-9f7c-4dfc-a688-822903dd79ba","Type":"ContainerStarted","Data":"82638661db2831415a9778723444fc4f8a170efc7e03a979e347fde1e1469ac2"} Jul 09 14:31:51.558629 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:51.558148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" event={"ID":"c8509b3b-9f7c-4dfc-a688-822903dd79ba","Type":"ContainerStarted","Data":"f408c9e6fbf40a8207fcd948a55fcf137e9682c835beb82995592de1b405cb5b"} Jul 09 14:31:51.558629 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:51.558200 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:31:51.607896 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:31:51.607840 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" podStartSLOduration=1.607823093 podStartE2EDuration="1.607823093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:31:51.605873565 +0000 UTC m=+1078.294802820" watchObservedRunningTime="2026-07-09 14:31:51.607823093 +0000 UTC m=+1078.296752331" Jul 09 14:32:02.565019 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:32:02.564989 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cmn45" Jul 09 14:33:53.849319 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:33:53.849284 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:33:53.859024 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:33:53.858997 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:37:40.945452 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.945416 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49"] Jul 09 14:37:40.949492 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.949476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:40.955371 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.955348 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5d40eff7-serving-cert\"" Jul 09 14:37:40.955827 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.955808 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bmdnz\"" Jul 09 14:37:40.967629 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.967596 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49"] Jul 09 14:37:40.978278 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.978244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:40.978452 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.978340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbrw\" (UniqueName: \"kubernetes.io/projected/61c47a9a-195b-4b7e-b24b-64ffb9c25816-kube-api-access-jhbrw\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:40.978526 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:40.978480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c47a9a-195b-4b7e-b24b-64ffb9c25816-openshift-service-ca\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.080021 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.079986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.080204 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.080039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbrw\" (UniqueName: \"kubernetes.io/projected/61c47a9a-195b-4b7e-b24b-64ffb9c25816-kube-api-access-jhbrw\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.080204 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.080061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c47a9a-195b-4b7e-b24b-64ffb9c25816-openshift-service-ca\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.080204 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:37:41.080130 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-serving-cert: secret "maas-api-e2e-ait-5d40eff7-serving-cert" not found Jul 09 14:37:41.080356 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:37:41.080206 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls podName:61c47a9a-195b-4b7e-b24b-64ffb9c25816 nodeName:}" failed. No retries permitted until 2026-07-09 14:37:41.580186214 +0000 UTC m=+1428.269115451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls") pod "maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" (UID: "61c47a9a-195b-4b7e-b24b-64ffb9c25816") : secret "maas-api-e2e-ait-5d40eff7-serving-cert" not found Jul 09 14:37:41.080717 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.080695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c47a9a-195b-4b7e-b24b-64ffb9c25816-openshift-service-ca\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.089838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.089815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbrw\" (UniqueName: \"kubernetes.io/projected/61c47a9a-195b-4b7e-b24b-64ffb9c25816-kube-api-access-jhbrw\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.584452 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.584412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.586849 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.586819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls\") pod \"maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.860484 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.860380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:41.999895 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:41.999868 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49"] Jul 09 14:37:42.002539 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:37:42.002507 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c47a9a_195b_4b7e_b24b_64ffb9c25816.slice/crio-727742c544e6734be6e0ac998b26fd9b5ce04bf4f6e4beacb561ace8ba33191a WatchSource:0}: Error finding container 727742c544e6734be6e0ac998b26fd9b5ce04bf4f6e4beacb561ace8ba33191a: Status 404 returned error can't find the container with id 727742c544e6734be6e0ac998b26fd9b5ce04bf4f6e4beacb561ace8ba33191a Jul 09 14:37:42.003975 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:42.003957 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:37:42.308002 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:42.307965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" event={"ID":"61c47a9a-195b-4b7e-b24b-64ffb9c25816","Type":"ContainerStarted","Data":"727742c544e6734be6e0ac998b26fd9b5ce04bf4f6e4beacb561ace8ba33191a"} Jul 09 14:37:45.326659 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:45.326623 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" event={"ID":"61c47a9a-195b-4b7e-b24b-64ffb9c25816","Type":"ContainerStarted","Data":"0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03"} Jul 09 14:37:45.327063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:45.326732 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:45.343143 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:45.343086 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" podStartSLOduration=2.844517823 podStartE2EDuration="5.343069557s" totalImagesPullingTime="2.498551734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:37:45.342109115 +0000 UTC m=+1432.031038381" watchObservedRunningTime="2026-07-09 14:37:45.343069557 +0000 UTC m=+1432.031998796" Jul 09 14:37:46.230275 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:46.230172 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49"] Jul 09 14:37:46.431298 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:37:46.431261 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-serving-cert: secret "maas-api-e2e-ait-5d40eff7-serving-cert" not found Jul 09 14:37:46.431687 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:37:46.431350 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls podName:61c47a9a-195b-4b7e-b24b-64ffb9c25816 nodeName:}" failed. No retries permitted until 2026-07-09 14:37:46.931326286 +0000 UTC m=+1433.620255523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls") pod "maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" (UID: "61c47a9a-195b-4b7e-b24b-64ffb9c25816") : secret "maas-api-e2e-ait-5d40eff7-serving-cert" not found Jul 09 14:37:46.935831 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:37:46.935797 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-serving-cert: secret "maas-api-e2e-ait-5d40eff7-serving-cert" not found Jul 09 14:37:46.936023 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:37:46.935865 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls podName:61c47a9a-195b-4b7e-b24b-64ffb9c25816 nodeName:}" failed. No retries permitted until 2026-07-09 14:37:47.935850476 +0000 UTC m=+1434.624779703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls") pod "maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" (UID: "61c47a9a-195b-4b7e-b24b-64ffb9c25816") : secret "maas-api-e2e-ait-5d40eff7-serving-cert" not found Jul 09 14:37:47.336955 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.336906 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" podUID="61c47a9a-195b-4b7e-b24b-64ffb9c25816" containerName="maas-api" containerID="cri-o://0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03" gracePeriod=30 Jul 09 14:37:47.592598 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.592525 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:47.643196 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.643167 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhbrw\" (UniqueName: \"kubernetes.io/projected/61c47a9a-195b-4b7e-b24b-64ffb9c25816-kube-api-access-jhbrw\") pod \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " Jul 09 14:37:47.643378 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.643208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c47a9a-195b-4b7e-b24b-64ffb9c25816-openshift-service-ca\") pod \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " Jul 09 14:37:47.643378 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.643327 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls\") pod \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\" (UID: \"61c47a9a-195b-4b7e-b24b-64ffb9c25816\") " Jul 09 14:37:47.643627 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.643602 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c47a9a-195b-4b7e-b24b-64ffb9c25816-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61c47a9a-195b-4b7e-b24b-64ffb9c25816" (UID: "61c47a9a-195b-4b7e-b24b-64ffb9c25816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:37:47.645423 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.645392 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61c47a9a-195b-4b7e-b24b-64ffb9c25816" (UID: "61c47a9a-195b-4b7e-b24b-64ffb9c25816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:37:47.645529 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.645502 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c47a9a-195b-4b7e-b24b-64ffb9c25816-kube-api-access-jhbrw" (OuterVolumeSpecName: "kube-api-access-jhbrw") pod "61c47a9a-195b-4b7e-b24b-64ffb9c25816" (UID: "61c47a9a-195b-4b7e-b24b-64ffb9c25816"). InnerVolumeSpecName "kube-api-access-jhbrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:37:47.744805 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.744761 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c47a9a-195b-4b7e-b24b-64ffb9c25816-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:37:47.744805 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.744798 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbrw\" (UniqueName: \"kubernetes.io/projected/61c47a9a-195b-4b7e-b24b-64ffb9c25816-kube-api-access-jhbrw\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:37:47.744805 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:47.744809 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c47a9a-195b-4b7e-b24b-64ffb9c25816-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:37:48.344958 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.344925 2575 generic.go:356] "Generic (PLEG): container finished" podID="61c47a9a-195b-4b7e-b24b-64ffb9c25816" containerID="0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03" exitCode=0 Jul 09 14:37:48.345147 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.344978 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" Jul 09 14:37:48.345147 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.344995 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" event={"ID":"61c47a9a-195b-4b7e-b24b-64ffb9c25816","Type":"ContainerDied","Data":"0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03"} Jul 09 14:37:48.345147 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.345048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49" event={"ID":"61c47a9a-195b-4b7e-b24b-64ffb9c25816","Type":"ContainerDied","Data":"727742c544e6734be6e0ac998b26fd9b5ce04bf4f6e4beacb561ace8ba33191a"} Jul 09 14:37:48.345147 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.345076 2575 scope.go:117] "RemoveContainer" containerID="0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03" Jul 09 14:37:48.355261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.355242 2575 scope.go:117] "RemoveContainer" containerID="0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03" Jul 09 14:37:48.355577 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:37:48.355551 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03\": container with ID starting with 0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03 not found: ID does not exist" containerID="0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03" Jul 09 14:37:48.355644 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.355591 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03"} err="failed to get container status \"0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03\": rpc error: code = NotFound desc = could not find container \"0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03\": container with ID starting with 0a628c203cc9f93b84c5eaa44269854cbd16db4bb34fd9f97f9efaf8550d9e03 not found: ID does not exist" Jul 09 14:37:48.376024 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.375988 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49"] Jul 09 14:37:48.381081 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:48.381054 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d40eff7-6946cbb7fd-rpz49"] Jul 09 14:37:49.886523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:49.886488 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c47a9a-195b-4b7e-b24b-64ffb9c25816" path="/var/lib/kubelet/pods/61c47a9a-195b-4b7e-b24b-64ffb9c25816/volumes" Jul 09 14:37:52.417105 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.417073 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w"] Jul 09 14:37:52.417515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.417500 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c47a9a-195b-4b7e-b24b-64ffb9c25816" containerName="maas-api" Jul 09 14:37:52.417515 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.417513 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c47a9a-195b-4b7e-b24b-64ffb9c25816" containerName="maas-api" Jul 09 14:37:52.417602 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.417585 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c47a9a-195b-4b7e-b24b-64ffb9c25816" containerName="maas-api" Jul 09 14:37:52.422101 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.422077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.425074 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.425048 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-899fd65b-openshift-default-dockercfg-6g786\"" Jul 09 14:37:52.425388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.425095 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 14:37:52.441835 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.441804 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w"] Jul 09 14:37:52.489132 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-certs\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489351 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967b5e04-427c-4492-b3d5-e8728621e720-istio-podinfo\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489351 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-istio-token\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489351 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-credential-socket\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-data\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-socket\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-envoy\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489560 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967b5e04-427c-4492-b3d5-e8728621e720-istiod-ca-cert\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.489560 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.489483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chsq\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-kube-api-access-9chsq\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.589905 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.589870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-data\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.589905 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.589906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-socket\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.589927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-envoy\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967b5e04-427c-4492-b3d5-e8728621e720-istiod-ca-cert\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chsq\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-kube-api-access-9chsq\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-certs\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590384 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967b5e04-427c-4492-b3d5-e8728621e720-istio-podinfo\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590384 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-istio-token\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590384 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-credential-socket\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590384 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-socket\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590622 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590499 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-data\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590684 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590648 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-certs\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590761 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-credential-socket\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.590991 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.590969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967b5e04-427c-4492-b3d5-e8728621e720-istiod-ca-cert\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.592739 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.592717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-envoy\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.593048 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.593023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967b5e04-427c-4492-b3d5-e8728621e720-istio-podinfo\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.607914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.607880 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-istio-token\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.611802 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.611773 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chsq\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-kube-api-access-9chsq\") pod \"e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.734469 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.734370 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:52.878668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:52.878637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w"] Jul 09 14:37:52.880264 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:37:52.880213 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967b5e04_427c_4492_b3d5_e8728621e720.slice/crio-26c33d4586054047c4a0531ceec1d983418604e0b1f02b7a8077e209598c53aa WatchSource:0}: Error finding container 26c33d4586054047c4a0531ceec1d983418604e0b1f02b7a8077e209598c53aa: Status 404 returned error can't find the container with id 26c33d4586054047c4a0531ceec1d983418604e0b1f02b7a8077e209598c53aa Jul 09 14:37:53.371232 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:53.371188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" event={"ID":"967b5e04-427c-4492-b3d5-e8728621e720","Type":"ContainerStarted","Data":"26c33d4586054047c4a0531ceec1d983418604e0b1f02b7a8077e209598c53aa"} Jul 09 14:37:56.204156 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:56.204118 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:37:56.204521 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:56.204201 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:37:56.204521 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:56.204248 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:37:56.391006 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:56.390970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" event={"ID":"967b5e04-427c-4492-b3d5-e8728621e720","Type":"ContainerStarted","Data":"89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef"} Jul 09 14:37:56.456342 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:56.456212 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" podStartSLOduration=1.134666104 podStartE2EDuration="4.456193783s" totalImagesPullingTime="3.321527679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:37:56.45027657 +0000 UTC m=+1443.139205825" watchObservedRunningTime="2026-07-09 14:37:56.456193783 +0000 UTC m=+1443.145123083" Jul 09 14:37:56.735523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:56.735420 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:57.740503 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:57.740470 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:58.400797 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:58.400756 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:37:58.401838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:37:58.401821 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:38:04.875666 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:04.875631 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w"] Jul 09 14:38:04.876104 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:04.875827 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" podUID="967b5e04-427c-4492-b3d5-e8728621e720" containerName="istio-proxy" containerID="cri-o://89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef" gracePeriod=30 Jul 09 14:38:06.030884 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.030858 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:38:06.120595 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120564 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-envoy\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.120773 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120622 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-certs\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.120773 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120642 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9chsq\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-kube-api-access-9chsq\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.120773 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120680 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-socket\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.120773 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120699 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-istio-token\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.120773 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120733 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967b5e04-427c-4492-b3d5-e8728621e720-istio-podinfo\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.121054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120796 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-data\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.121054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120851 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-credential-socket\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.121054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120877 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967b5e04-427c-4492-b3d5-e8728621e720-istiod-ca-cert\") pod \"967b5e04-427c-4492-b3d5-e8728621e720\" (UID: \"967b5e04-427c-4492-b3d5-e8728621e720\") " Jul 09 14:38:06.121054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.120994 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:06.121054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.121008 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:06.121349 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.121195 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.121349 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.121216 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.121349 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.121288 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:06.121349 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.121317 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967b5e04-427c-4492-b3d5-e8728621e720-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:38:06.121996 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.121962 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-data" (OuterVolumeSpecName: "istio-data") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:06.122913 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.122882 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:06.123276 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.123255 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-istio-token" (OuterVolumeSpecName: "istio-token") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:38:06.123355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.123322 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/967b5e04-427c-4492-b3d5-e8728621e720-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:38:06.123395 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.123365 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-kube-api-access-9chsq" (OuterVolumeSpecName: "kube-api-access-9chsq") pod "967b5e04-427c-4492-b3d5-e8728621e720" (UID: "967b5e04-427c-4492-b3d5-e8728621e720"). InnerVolumeSpecName "kube-api-access-9chsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:38:06.221811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.221760 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.221811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.221805 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.221811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.221815 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967b5e04-427c-4492-b3d5-e8728621e720-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.221811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.221824 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967b5e04-427c-4492-b3d5-e8728621e720-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.222135 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.221834 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9chsq\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-kube-api-access-9chsq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.222135 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.221842 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967b5e04-427c-4492-b3d5-e8728621e720-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.222135 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.221850 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967b5e04-427c-4492-b3d5-e8728621e720-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:06.446316 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.446202 2575 generic.go:356] "Generic (PLEG): container finished" podID="967b5e04-427c-4492-b3d5-e8728621e720" containerID="89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef" exitCode=0 Jul 09 14:38:06.446316 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.446275 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" Jul 09 14:38:06.446316 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.446294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" event={"ID":"967b5e04-427c-4492-b3d5-e8728621e720","Type":"ContainerDied","Data":"89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef"} Jul 09 14:38:06.446532 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.446349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w" event={"ID":"967b5e04-427c-4492-b3d5-e8728621e720","Type":"ContainerDied","Data":"26c33d4586054047c4a0531ceec1d983418604e0b1f02b7a8077e209598c53aa"} Jul 09 14:38:06.446532 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.446372 2575 scope.go:117] "RemoveContainer" containerID="89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef" Jul 09 14:38:06.456184 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.456164 2575 scope.go:117] "RemoveContainer" containerID="89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef" Jul 09 14:38:06.456502 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:38:06.456483 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef\": container with ID starting with 89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef not found: ID does not exist" containerID="89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef" Jul 09 14:38:06.456574 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.456511 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef"} err="failed to get container status \"89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef\": rpc error: code = NotFound desc = could not find container \"89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef\": container with ID starting with 89aef40e8fd70eea11012f7d73247265597e564dbcb2c2e0cde4aab4191376ef not found: ID does not exist" Jul 09 14:38:06.483630 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.483596 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w"] Jul 09 14:38:06.494556 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:06.494525 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-899fd65b-openshift-default-6fdc5b7989-tth6w"] Jul 09 14:38:07.886432 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:07.886399 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="967b5e04-427c-4492-b3d5-e8728621e720" path="/var/lib/kubelet/pods/967b5e04-427c-4492-b3d5-e8728621e720/volumes" Jul 09 14:38:12.582874 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.582835 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs"] Jul 09 14:38:12.583273 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.583217 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967b5e04-427c-4492-b3d5-e8728621e720" containerName="istio-proxy" Jul 09 14:38:12.583273 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.583244 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="967b5e04-427c-4492-b3d5-e8728621e720" containerName="istio-proxy" Jul 09 14:38:12.583365 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.583331 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="967b5e04-427c-4492-b3d5-e8728621e720" containerName="istio-proxy" Jul 09 14:38:12.590184 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.590160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.596148 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.596122 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-270537f6-openshift-default-dockercfg-kwzrn\"" Jul 09 14:38:12.596297 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.596146 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 14:38:12.615523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.615488 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs"] Jul 09 14:38:12.679656 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-data\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.679825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-certs\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.679825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-podinfo\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.679825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-credential-socket\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.679825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e83c8-a050-4635-baba-779ab9d5a27f-istiod-ca-cert\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.679825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllbz\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-kube-api-access-vllbz\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.680006 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-token\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.680006 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-socket\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.680006 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.679883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-envoy\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780317 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-data\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780317 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-certs\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-podinfo\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-credential-socket\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e83c8-a050-4635-baba-779ab9d5a27f-istiod-ca-cert\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllbz\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-kube-api-access-vllbz\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-token\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-socket\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-envoy\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780699 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-credential-socket\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.780767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-certs\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.781026 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.780764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-data\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.781097 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.781061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-socket\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.781285 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.781264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e83c8-a050-4635-baba-779ab9d5a27f-istiod-ca-cert\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.782856 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.782827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-envoy\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.782957 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.782906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-podinfo\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.809921 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.809844 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllbz\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-kube-api-access-vllbz\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.813073 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.813050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-token\") pod \"e2e-derive-270537f6-openshift-default-54477cc975-xxwfs\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:12.901151 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:12.901114 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:13.096580 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.096554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs"] Jul 09 14:38:13.098703 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:38:13.098678 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2e83c8_a050_4635_baba_779ab9d5a27f.slice/crio-703fcf29e576e1c15adcbb67afdb0a58e36d68e9ca11cb8264a1cfd411480ce2 WatchSource:0}: Error finding container 703fcf29e576e1c15adcbb67afdb0a58e36d68e9ca11cb8264a1cfd411480ce2: Status 404 returned error can't find the container with id 703fcf29e576e1c15adcbb67afdb0a58e36d68e9ca11cb8264a1cfd411480ce2 Jul 09 14:38:13.101751 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.101719 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:38:13.101856 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.101794 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:38:13.101856 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.101840 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:38:13.485008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.484905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" event={"ID":"ca2e83c8-a050-4635-baba-779ab9d5a27f","Type":"ContainerStarted","Data":"97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd"} Jul 09 14:38:13.485008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.484966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" event={"ID":"ca2e83c8-a050-4635-baba-779ab9d5a27f","Type":"ContainerStarted","Data":"703fcf29e576e1c15adcbb67afdb0a58e36d68e9ca11cb8264a1cfd411480ce2"} Jul 09 14:38:13.522061 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.522008 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" podStartSLOduration=1.521988758 podStartE2EDuration="1.521988758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:38:13.518976424 +0000 UTC m=+1460.207905662" watchObservedRunningTime="2026-07-09 14:38:13.521988758 +0000 UTC m=+1460.210917997" Jul 09 14:38:13.902087 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:13.902058 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:14.907393 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:14.907356 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:15.495559 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:15.495524 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:15.496559 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:15.496538 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:18.275006 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:18.274965 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs"] Jul 09 14:38:18.509206 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:18.509154 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" podUID="ca2e83c8-a050-4635-baba-779ab9d5a27f" containerName="istio-proxy" containerID="cri-o://97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd" gracePeriod=30 Jul 09 14:38:23.769952 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.769923 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:23.883056 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883021 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-token\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883056 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883060 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vllbz\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-kube-api-access-vllbz\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883326 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883087 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-data\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883326 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883108 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e83c8-a050-4635-baba-779ab9d5a27f-istiod-ca-cert\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883326 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883286 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-credential-socket\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883483 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883382 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-certs\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883483 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:23.883483 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883455 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-socket\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883483 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883479 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-envoy\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883693 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883511 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-podinfo\") pod \"ca2e83c8-a050-4635-baba-779ab9d5a27f\" (UID: \"ca2e83c8-a050-4635-baba-779ab9d5a27f\") " Jul 09 14:38:23.883693 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883603 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2e83c8-a050-4635-baba-779ab9d5a27f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:38:23.883806 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883729 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:23.883873 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883848 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.884008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883868 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.884008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.884006 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e83c8-a050-4635-baba-779ab9d5a27f-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.884151 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883935 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:23.884151 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.883961 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:23.885621 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.885572 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:38:23.885738 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.885627 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:38:23.885738 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.885643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:38:23.885854 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.885805 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-kube-api-access-vllbz" (OuterVolumeSpecName: "kube-api-access-vllbz") pod "ca2e83c8-a050-4635-baba-779ab9d5a27f" (UID: "ca2e83c8-a050-4635-baba-779ab9d5a27f"). InnerVolumeSpecName "kube-api-access-vllbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:38:23.984778 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.984741 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.984778 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.984766 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.984778 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.984775 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.984778 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.984783 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.985063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.984791 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllbz\" (UniqueName: \"kubernetes.io/projected/ca2e83c8-a050-4635-baba-779ab9d5a27f-kube-api-access-vllbz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:23.985063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:23.984801 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e83c8-a050-4635-baba-779ab9d5a27f-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:38:24.541082 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.541050 2575 generic.go:356] "Generic (PLEG): container finished" podID="ca2e83c8-a050-4635-baba-779ab9d5a27f" containerID="97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd" exitCode=0 Jul 09 14:38:24.541275 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.541096 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" Jul 09 14:38:24.541275 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.541135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" event={"ID":"ca2e83c8-a050-4635-baba-779ab9d5a27f","Type":"ContainerDied","Data":"97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd"} Jul 09 14:38:24.541275 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.541188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs" event={"ID":"ca2e83c8-a050-4635-baba-779ab9d5a27f","Type":"ContainerDied","Data":"703fcf29e576e1c15adcbb67afdb0a58e36d68e9ca11cb8264a1cfd411480ce2"} Jul 09 14:38:24.541275 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.541212 2575 scope.go:117] "RemoveContainer" containerID="97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd" Jul 09 14:38:24.553871 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.553845 2575 scope.go:117] "RemoveContainer" containerID="97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd" Jul 09 14:38:24.554173 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:38:24.554152 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd\": container with ID starting with 97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd not found: ID does not exist" containerID="97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd" Jul 09 14:38:24.554300 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.554182 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd"} err="failed to get container status \"97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd\": rpc error: code = NotFound desc = could not find container \"97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd\": container with ID starting with 97412ee7b98efff6351e0f95480ffe96da6d48dc7bae8d8a867d02eb0ed04ccd not found: ID does not exist" Jul 09 14:38:24.571512 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.571472 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs"] Jul 09 14:38:24.573185 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:24.573162 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-270537f6-openshift-default-54477cc975-xxwfs"] Jul 09 14:38:25.886998 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:25.886965 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca2e83c8-a050-4635-baba-779ab9d5a27f" path="/var/lib/kubelet/pods/ca2e83c8-a050-4635-baba-779ab9d5a27f/volumes" Jul 09 14:38:53.883735 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:53.883703 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:38:53.902060 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:38:53.902031 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:40:08.134099 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.134061 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs"] Jul 09 14:40:08.134746 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.134459 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2e83c8-a050-4635-baba-779ab9d5a27f" containerName="istio-proxy" Jul 09 14:40:08.134746 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.134471 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2e83c8-a050-4635-baba-779ab9d5a27f" containerName="istio-proxy" Jul 09 14:40:08.134746 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.134539 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2e83c8-a050-4635-baba-779ab9d5a27f" containerName="istio-proxy" Jul 09 14:40:08.137802 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.137781 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.141926 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.141904 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a0d9b122-serving-cert\"" Jul 09 14:40:08.142103 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.142006 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bmdnz\"" Jul 09 14:40:08.169834 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.169801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs"] Jul 09 14:40:08.215095 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.215061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gvp\" (UniqueName: \"kubernetes.io/projected/0f6ca67e-1378-4c32-9416-44a2e30cfecc-kube-api-access-g9gvp\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.215272 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.215122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ca67e-1378-4c32-9416-44a2e30cfecc-maas-api-tls\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.215272 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.215186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ca67e-1378-4c32-9416-44a2e30cfecc-openshift-service-ca\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.315899 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.315862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gvp\" (UniqueName: \"kubernetes.io/projected/0f6ca67e-1378-4c32-9416-44a2e30cfecc-kube-api-access-g9gvp\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.316080 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.315926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ca67e-1378-4c32-9416-44a2e30cfecc-maas-api-tls\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.316080 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.315965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ca67e-1378-4c32-9416-44a2e30cfecc-openshift-service-ca\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.316689 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.316665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ca67e-1378-4c32-9416-44a2e30cfecc-openshift-service-ca\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.318431 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.318410 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ca67e-1378-4c32-9416-44a2e30cfecc-maas-api-tls\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.330446 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.330421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gvp\" (UniqueName: \"kubernetes.io/projected/0f6ca67e-1378-4c32-9416-44a2e30cfecc-kube-api-access-g9gvp\") pod \"maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.449824 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.449726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:08.633841 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:40:08.633814 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6ca67e_1378_4c32_9416_44a2e30cfecc.slice/crio-55bcc27013bf99012d739dfbb34d05f80a150a8b27ec0a959b03f49c02a182d5 WatchSource:0}: Error finding container 55bcc27013bf99012d739dfbb34d05f80a150a8b27ec0a959b03f49c02a182d5: Status 404 returned error can't find the container with id 55bcc27013bf99012d739dfbb34d05f80a150a8b27ec0a959b03f49c02a182d5 Jul 09 14:40:08.634363 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:08.634339 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs"] Jul 09 14:40:09.077522 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:09.077476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" event={"ID":"0f6ca67e-1378-4c32-9416-44a2e30cfecc","Type":"ContainerStarted","Data":"55bcc27013bf99012d739dfbb34d05f80a150a8b27ec0a959b03f49c02a182d5"} Jul 09 14:40:10.083510 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:10.083471 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" event={"ID":"0f6ca67e-1378-4c32-9416-44a2e30cfecc","Type":"ContainerStarted","Data":"104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9"} Jul 09 14:40:10.083927 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:10.083599 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:10.103767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:10.103685 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" podStartSLOduration=1.655802043 podStartE2EDuration="2.103666058s" totalImagesPullingTime="447.864015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:10.102761333 +0000 UTC m=+1576.791690589" watchObservedRunningTime="2026-07-09 14:40:10.103666058 +0000 UTC m=+1576.792595296" Jul 09 14:40:16.094336 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:16.094306 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:20.188712 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.188673 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs"] Jul 09 14:40:20.189249 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.188956 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" podUID="0f6ca67e-1378-4c32-9416-44a2e30cfecc" containerName="maas-api" containerID="cri-o://104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9" gracePeriod=30 Jul 09 14:40:20.446123 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.446048 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:20.531084 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.531050 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9gvp\" (UniqueName: \"kubernetes.io/projected/0f6ca67e-1378-4c32-9416-44a2e30cfecc-kube-api-access-g9gvp\") pod \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " Jul 09 14:40:20.531330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.531106 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ca67e-1378-4c32-9416-44a2e30cfecc-maas-api-tls\") pod \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " Jul 09 14:40:20.531330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.531164 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ca67e-1378-4c32-9416-44a2e30cfecc-openshift-service-ca\") pod \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\" (UID: \"0f6ca67e-1378-4c32-9416-44a2e30cfecc\") " Jul 09 14:40:20.531558 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.531530 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6ca67e-1378-4c32-9416-44a2e30cfecc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f6ca67e-1378-4c32-9416-44a2e30cfecc" (UID: "0f6ca67e-1378-4c32-9416-44a2e30cfecc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:40:20.533297 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.533267 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6ca67e-1378-4c32-9416-44a2e30cfecc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f6ca67e-1378-4c32-9416-44a2e30cfecc" (UID: "0f6ca67e-1378-4c32-9416-44a2e30cfecc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:40:20.533297 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.533277 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6ca67e-1378-4c32-9416-44a2e30cfecc-kube-api-access-g9gvp" (OuterVolumeSpecName: "kube-api-access-g9gvp") pod "0f6ca67e-1378-4c32-9416-44a2e30cfecc" (UID: "0f6ca67e-1378-4c32-9416-44a2e30cfecc"). InnerVolumeSpecName "kube-api-access-g9gvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:40:20.632658 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.632620 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ca67e-1378-4c32-9416-44a2e30cfecc-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:40:20.632658 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.632652 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ca67e-1378-4c32-9416-44a2e30cfecc-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:40:20.632658 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:20.632662 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9gvp\" (UniqueName: \"kubernetes.io/projected/0f6ca67e-1378-4c32-9416-44a2e30cfecc-kube-api-access-g9gvp\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:40:21.140517 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.140483 2575 generic.go:356] "Generic (PLEG): container finished" podID="0f6ca67e-1378-4c32-9416-44a2e30cfecc" containerID="104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9" exitCode=0 Jul 09 14:40:21.140724 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.140523 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" Jul 09 14:40:21.140724 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.140565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" event={"ID":"0f6ca67e-1378-4c32-9416-44a2e30cfecc","Type":"ContainerDied","Data":"104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9"} Jul 09 14:40:21.140724 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.140624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs" event={"ID":"0f6ca67e-1378-4c32-9416-44a2e30cfecc","Type":"ContainerDied","Data":"55bcc27013bf99012d739dfbb34d05f80a150a8b27ec0a959b03f49c02a182d5"} Jul 09 14:40:21.140724 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.140653 2575 scope.go:117] "RemoveContainer" containerID="104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9" Jul 09 14:40:21.150621 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.150598 2575 scope.go:117] "RemoveContainer" containerID="104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9" Jul 09 14:40:21.150902 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:40:21.150880 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9\": container with ID starting with 104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9 not found: ID does not exist" containerID="104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9" Jul 09 14:40:21.150970 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.150913 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9"} err="failed to get container status \"104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9\": rpc error: code = NotFound desc = could not find container \"104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9\": container with ID starting with 104c5c5aac1d4d39fd1f201f507aff42ae2ff47de2d30372406d137d3fb8cca9 not found: ID does not exist" Jul 09 14:40:21.163201 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.163165 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs"] Jul 09 14:40:21.169749 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.169721 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0d9b122-568678d8d7-qd4xs"] Jul 09 14:40:21.886707 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:21.886675 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6ca67e-1378-4c32-9416-44a2e30cfecc" path="/var/lib/kubelet/pods/0f6ca67e-1378-4c32-9416-44a2e30cfecc/volumes" Jul 09 14:40:46.201126 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.201038 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d"] Jul 09 14:40:46.201817 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.201486 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6ca67e-1378-4c32-9416-44a2e30cfecc" containerName="maas-api" Jul 09 14:40:46.201817 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.201500 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6ca67e-1378-4c32-9416-44a2e30cfecc" containerName="maas-api" Jul 09 14:40:46.201817 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.201586 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6ca67e-1378-4c32-9416-44a2e30cfecc" containerName="maas-api" Jul 09 14:40:46.205151 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.205130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.207668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.207644 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2a4ea46b-openshift-default-dockercfg-vdkhh\"" Jul 09 14:40:46.207786 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.207656 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 14:40:46.217421 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.217396 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d"] Jul 09 14:40:46.257591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-credential-socket\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257591 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-kube-api-access-pxt76\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-token\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-envoy\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-podinfo\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istiod-ca-cert\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-data\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257986 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257927 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-socket\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.257986 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.257964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-certs\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.359457 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-credential-socket\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.359667 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-kube-api-access-pxt76\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.359667 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-token\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.359667 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-envoy\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.359667 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-podinfo\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.359667 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istiod-ca-cert\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.359667 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-data\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.360023 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-socket\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.360023 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359893 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-credential-socket\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.360023 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.359927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-certs\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.360186 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.360090 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-data\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.360186 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.360146 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-socket\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.360301 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.360246 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-certs\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.360688 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.360667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istiod-ca-cert\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.362427 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.362404 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-envoy\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.362685 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.362665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-podinfo\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.369855 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.369833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-kube-api-access-pxt76\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.371912 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.371891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-token\") pod \"e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.516542 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.516449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:46.662727 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.662692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d"] Jul 09 14:40:46.665975 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:40:46.665949 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e0db73_a3f3_4067_bf34_39b73d2b286d.slice/crio-6b25ae70b3364c500450f4a1b7de59c594e3fd433e4f10425a94c1aae1930ac1 WatchSource:0}: Error finding container 6b25ae70b3364c500450f4a1b7de59c594e3fd433e4f10425a94c1aae1930ac1: Status 404 returned error can't find the container with id 6b25ae70b3364c500450f4a1b7de59c594e3fd433e4f10425a94c1aae1930ac1 Jul 09 14:40:46.668095 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.668050 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:40:46.668211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.668150 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:40:46.668211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:46.668197 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:40:47.272942 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:47.272909 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" event={"ID":"d4e0db73-a3f3-4067-bf34-39b73d2b286d","Type":"ContainerStarted","Data":"ae835fc36974fc25a7baea527dcb05577f8a7e5ea886530ebc73c1312e81ff7e"} Jul 09 14:40:47.273475 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:47.272960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" event={"ID":"d4e0db73-a3f3-4067-bf34-39b73d2b286d","Type":"ContainerStarted","Data":"6b25ae70b3364c500450f4a1b7de59c594e3fd433e4f10425a94c1aae1930ac1"} Jul 09 14:40:47.295863 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:47.295808 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podStartSLOduration=1.295788768 podStartE2EDuration="1.295788768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:47.294759048 +0000 UTC m=+1613.983688288" watchObservedRunningTime="2026-07-09 14:40:47.295788768 +0000 UTC m=+1613.984718008" Jul 09 14:40:47.517315 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:47.517278 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:47.522123 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:47.522099 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:48.277965 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:48.277937 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:40:48.278854 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:40:48.278833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:41:03.280286 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.280250 2575 patch_prober.go:28] interesting pod/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:41:03.280700 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.280320 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:41:03.356610 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.356520 2575 generic.go:356] "Generic (PLEG): container finished" podID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerID="ae835fc36974fc25a7baea527dcb05577f8a7e5ea886530ebc73c1312e81ff7e" exitCode=0 Jul 09 14:41:03.356610 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.356574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" event={"ID":"d4e0db73-a3f3-4067-bf34-39b73d2b286d","Type":"ContainerDied","Data":"ae835fc36974fc25a7baea527dcb05577f8a7e5ea886530ebc73c1312e81ff7e"} Jul 09 14:41:03.356962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.356946 2575 scope.go:117] "RemoveContainer" containerID="ae835fc36974fc25a7baea527dcb05577f8a7e5ea886530ebc73c1312e81ff7e" Jul 09 14:41:03.357944 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.357913 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:41:03.358019 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.357995 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:41:03.358060 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:03.358040 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:41:04.363917 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:04.363882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" event={"ID":"d4e0db73-a3f3-4067-bf34-39b73d2b286d","Type":"ContainerStarted","Data":"8d55bb4e65bd171f42d438f18238cc09ff644f489aa23092360949aa86031117"} Jul 09 14:41:04.516826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:04.516782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:41:04.517091 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:04.517066 2575 patch_prober.go:28] interesting pod/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:41:04.517141 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:04.517119 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:41:05.517124 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:05.517084 2575 patch_prober.go:28] interesting pod/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:41:05.517560 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:05.517157 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:41:06.516953 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:06.516919 2575 patch_prober.go:28] interesting pod/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:41:06.517152 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:06.516976 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:41:07.517205 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:07.517170 2575 patch_prober.go:28] interesting pod/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:41:07.517626 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:07.517253 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:41:08.517627 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:08.517585 2575 patch_prober.go:28] interesting pod/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:41:08.518039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:08.517659 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:41:09.521209 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:09.521178 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:41:09.521611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:09.521446 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:41:09.522024 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:41:09.522004 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:42:15.576147 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:15.576114 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d"] Jul 09 14:42:15.577019 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:15.576390 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" containerID="cri-o://8d55bb4e65bd171f42d438f18238cc09ff644f489aa23092360949aa86031117" gracePeriod=30 Jul 09 14:42:20.743445 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.743413 2575 generic.go:356] "Generic (PLEG): container finished" podID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerID="8d55bb4e65bd171f42d438f18238cc09ff644f489aa23092360949aa86031117" exitCode=0 Jul 09 14:42:20.743848 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.743457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" event={"ID":"d4e0db73-a3f3-4067-bf34-39b73d2b286d","Type":"ContainerDied","Data":"8d55bb4e65bd171f42d438f18238cc09ff644f489aa23092360949aa86031117"} Jul 09 14:42:20.743848 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.743538 2575 scope.go:117] "RemoveContainer" containerID="ae835fc36974fc25a7baea527dcb05577f8a7e5ea886530ebc73c1312e81ff7e" Jul 09 14:42:20.833852 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.833828 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:42:20.930047 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.929950 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-data\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930047 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.929993 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-podinfo\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930047 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930023 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-socket\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-kube-api-access-pxt76\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930101 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istiod-ca-cert\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-certs\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930161 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-token\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930198 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-envoy\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930253 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-credential-socket\") pod \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\" (UID: \"d4e0db73-a3f3-4067-bf34-39b73d2b286d\") " Jul 09 14:42:20.930681 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930487 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:42:20.930681 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930565 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:42:20.930681 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930578 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:20.930855 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930707 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:42:20.930923 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930857 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:42:20.931001 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.930983 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:42:20.932389 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.932366 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:42:20.932487 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.932406 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-kube-api-access-pxt76" (OuterVolumeSpecName: "kube-api-access-pxt76") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "kube-api-access-pxt76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:42:20.932655 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.932632 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:42:20.932758 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:20.932725 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4e0db73-a3f3-4067-bf34-39b73d2b286d" (UID: "d4e0db73-a3f3-4067-bf34-39b73d2b286d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:42:21.032022 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.031964 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxt76\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-kube-api-access-pxt76\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.032022 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.032020 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.032022 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.032030 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.032022 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.032039 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.032331 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.032047 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.032331 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.032055 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.032331 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.032064 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e0db73-a3f3-4067-bf34-39b73d2b286d-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.032331 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.032071 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e0db73-a3f3-4067-bf34-39b73d2b286d-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:42:21.749298 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.749260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" event={"ID":"d4e0db73-a3f3-4067-bf34-39b73d2b286d","Type":"ContainerDied","Data":"6b25ae70b3364c500450f4a1b7de59c594e3fd433e4f10425a94c1aae1930ac1"} Jul 09 14:42:21.749774 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.749327 2575 scope.go:117] "RemoveContainer" containerID="8d55bb4e65bd171f42d438f18238cc09ff644f489aa23092360949aa86031117" Jul 09 14:42:21.749774 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.749273 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d" Jul 09 14:42:21.774322 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.774291 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d"] Jul 09 14:42:21.779531 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.779506 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2a4ea46b-openshift-default-8449dc58dc-jsz6d"] Jul 09 14:42:21.894319 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:42:21.894277 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" path="/var/lib/kubelet/pods/d4e0db73-a3f3-4067-bf34-39b73d2b286d/volumes" Jul 09 14:43:17.894318 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.894279 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs"] Jul 09 14:43:17.894842 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.894817 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" Jul 09 14:43:17.894902 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.894843 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" Jul 09 14:43:17.894977 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.894965 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" Jul 09 14:43:17.895019 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.894981 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" Jul 09 14:43:17.898580 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.898563 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.901057 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.901032 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 14:43:17.901160 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.901032 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4777cd-openshift-default-dockercfg-zq84p\"" Jul 09 14:43:17.910054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.910030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs"] Jul 09 14:43:17.940073 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmr7h\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-kube-api-access-cmr7h\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940302 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-envoy\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940302 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-istio-token\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940302 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-credential-socket\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940302 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940268 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd51e30c-4060-4753-815e-620823a9fcdd-istiod-ca-cert\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940572 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-socket\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940633 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd51e30c-4060-4753-815e-620823a9fcdd-istio-podinfo\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940633 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-certs\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:17.940744 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:17.940656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-data\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.041807 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.041758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-socket\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.041807 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.041810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd51e30c-4060-4753-815e-620823a9fcdd-istio-podinfo\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.041943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-certs\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042063 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-data\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042172 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmr7h\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-kube-api-access-cmr7h\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042172 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-envoy\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042172 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-istio-token\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042359 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-credential-socket\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042359 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042267 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-socket\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042463 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-data\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042521 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-certs\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042521 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd51e30c-4060-4753-815e-620823a9fcdd-istiod-ca-cert\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.042796 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.042776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-credential-socket\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.043052 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.043029 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd51e30c-4060-4753-815e-620823a9fcdd-istiod-ca-cert\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.044426 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.044403 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-envoy\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.044743 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.044722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd51e30c-4060-4753-815e-620823a9fcdd-istio-podinfo\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.050241 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.050208 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-istio-token\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.050355 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.050330 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmr7h\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-kube-api-access-cmr7h\") pod \"e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.212685 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.212584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:18.355872 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.354959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs"] Jul 09 14:43:18.359539 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.359503 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:43:18.359669 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.359577 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:43:18.359669 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:18.359606 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:43:19.037869 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:19.037832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" event={"ID":"bd51e30c-4060-4753-815e-620823a9fcdd","Type":"ContainerStarted","Data":"0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb"} Jul 09 14:43:19.038302 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:19.037903 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" event={"ID":"bd51e30c-4060-4753-815e-620823a9fcdd","Type":"ContainerStarted","Data":"2fd95bbcac9b97ade90d0e55d0e863a55dbf172fc21c9c14f9e7203e44fcce54"} Jul 09 14:43:19.059074 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:19.059023 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podStartSLOduration=2.05900813 podStartE2EDuration="2.05900813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:43:19.057696409 +0000 UTC m=+1765.746625674" watchObservedRunningTime="2026-07-09 14:43:19.05900813 +0000 UTC m=+1765.747937369" Jul 09 14:43:19.213753 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:19.213706 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:19.218728 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:19.218699 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:20.043463 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:20.043430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:20.044442 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:20.044420 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:21.019754 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.019720 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485"] Jul 09 14:43:21.020134 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.020122 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" Jul 09 14:43:21.020179 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.020135 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e0db73-a3f3-4067-bf34-39b73d2b286d" containerName="istio-proxy" Jul 09 14:43:21.023538 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.023518 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.027325 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.027301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bmdnz\"" Jul 09 14:43:21.027750 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.027507 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4777cd-serving-cert\"" Jul 09 14:43:21.039040 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.039013 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485"] Jul 09 14:43:21.072204 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.072170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.072618 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.072259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.072618 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.072280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8dw\" (UniqueName: \"kubernetes.io/projected/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-kube-api-access-jx8dw\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.173152 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.173114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.173361 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:43:21.173279 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-serving-cert: secret "maas-api-e2e-inf-b-4777cd-serving-cert" not found Jul 09 14:43:21.173361 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:43:21.173338 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls podName:40fed4cd-b8ee-4822-9f81-c7f2d299f5c5 nodeName:}" failed. No retries permitted until 2026-07-09 14:43:21.6733164 +0000 UTC m=+1768.362245617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls") pod "maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" (UID: "40fed4cd-b8ee-4822-9f81-c7f2d299f5c5") : secret "maas-api-e2e-inf-b-4777cd-serving-cert" not found Jul 09 14:43:21.173501 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.173359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.173501 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.173393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8dw\" (UniqueName: \"kubernetes.io/projected/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-kube-api-access-jx8dw\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.174048 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.174028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.186069 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.186038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8dw\" (UniqueName: \"kubernetes.io/projected/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-kube-api-access-jx8dw\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.678167 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.678124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.680649 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.680624 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls\") pod \"maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:21.935489 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:21.935386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:22.081486 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:22.081458 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485"] Jul 09 14:43:22.083929 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:43:22.083886 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fed4cd_b8ee_4822_9f81_c7f2d299f5c5.slice/crio-eb2e5f6068694522a262e1c2e90af4f8862c6097fef7d99e64dcb685b232c416 WatchSource:0}: Error finding container eb2e5f6068694522a262e1c2e90af4f8862c6097fef7d99e64dcb685b232c416: Status 404 returned error can't find the container with id eb2e5f6068694522a262e1c2e90af4f8862c6097fef7d99e64dcb685b232c416 Jul 09 14:43:22.085702 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:22.085684 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:43:23.063166 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:23.063127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" event={"ID":"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5","Type":"ContainerStarted","Data":"ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120"} Jul 09 14:43:23.063404 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:23.063251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" event={"ID":"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5","Type":"ContainerStarted","Data":"eb2e5f6068694522a262e1c2e90af4f8862c6097fef7d99e64dcb685b232c416"} Jul 09 14:43:23.063404 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:23.063285 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:23.081296 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:23.081240 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" podStartSLOduration=2.640311507 podStartE2EDuration="3.081200634s" totalImagesPullingTime="440.889127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:43:23.079636363 +0000 UTC m=+1769.768565627" watchObservedRunningTime="2026-07-09 14:43:23.081200634 +0000 UTC m=+1769.770129872" Jul 09 14:43:29.078495 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:29.078467 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:43:35.045195 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.045151 2575 patch_prober.go:28] interesting pod/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 14:43:35.045633 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.045254 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 14:43:35.138946 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.138910 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd51e30c-4060-4753-815e-620823a9fcdd" containerID="0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb" exitCode=0 Jul 09 14:43:35.139123 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.138996 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" event={"ID":"bd51e30c-4060-4753-815e-620823a9fcdd","Type":"ContainerDied","Data":"0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb"} Jul 09 14:43:35.139513 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.139496 2575 scope.go:117] "RemoveContainer" containerID="0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb" Jul 09 14:43:35.140775 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.140740 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:43:35.140875 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.140834 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:43:35.140916 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:35.140880 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:43:36.150334 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:36.150292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" event={"ID":"bd51e30c-4060-4753-815e-620823a9fcdd","Type":"ContainerStarted","Data":"fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c"} Jul 09 14:43:36.213779 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:36.213747 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:36.214013 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:36.213986 2575 patch_prober.go:28] interesting pod/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 14:43:36.214073 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:36.214053 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 14:43:37.213441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:37.213408 2575 patch_prober.go:28] interesting pod/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 14:43:37.213865 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:37.213462 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 14:43:38.213574 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:38.213535 2575 patch_prober.go:28] interesting pod/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 14:43:38.213988 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:38.213596 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 14:43:39.213042 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:39.212998 2575 patch_prober.go:28] interesting pod/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 14:43:39.213254 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:39.213052 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 14:43:40.217166 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:40.217127 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:40.217625 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:40.217490 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:40.218182 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:40.218166 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:43:53.924954 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:53.924926 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:43:53.934477 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:43:53.934450 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:45:00.152851 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.152813 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq"] Jul 09 14:45:00.156265 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.156246 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p"] Jul 09 14:45:00.156336 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.156302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" Jul 09 14:45:00.159735 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.159715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" Jul 09 14:45:00.168576 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.168543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq"] Jul 09 14:45:00.171607 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.171570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p"] Jul 09 14:45:00.298729 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.298690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5bq\" (UniqueName: \"kubernetes.io/projected/e07fa04a-8ab0-4ca5-b401-8ae31d2653de-kube-api-access-ld5bq\") pod \"maas-api-key-cleanup-29726805-t9rtq\" (UID: \"e07fa04a-8ab0-4ca5-b401-8ae31d2653de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" Jul 09 14:45:00.298930 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.298814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdb6\" (UniqueName: \"kubernetes.io/projected/6ac3dc7a-adba-4127-8023-917efc329045-kube-api-access-pzdb6\") pod \"maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p\" (UID: \"6ac3dc7a-adba-4127-8023-917efc329045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" Jul 09 14:45:00.400302 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.400257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdb6\" (UniqueName: \"kubernetes.io/projected/6ac3dc7a-adba-4127-8023-917efc329045-kube-api-access-pzdb6\") pod \"maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p\" (UID: \"6ac3dc7a-adba-4127-8023-917efc329045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" Jul 09 14:45:00.400511 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.400315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5bq\" (UniqueName: \"kubernetes.io/projected/e07fa04a-8ab0-4ca5-b401-8ae31d2653de-kube-api-access-ld5bq\") pod \"maas-api-key-cleanup-29726805-t9rtq\" (UID: \"e07fa04a-8ab0-4ca5-b401-8ae31d2653de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" Jul 09 14:45:00.411068 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.410384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5bq\" (UniqueName: \"kubernetes.io/projected/e07fa04a-8ab0-4ca5-b401-8ae31d2653de-kube-api-access-ld5bq\") pod \"maas-api-key-cleanup-29726805-t9rtq\" (UID: \"e07fa04a-8ab0-4ca5-b401-8ae31d2653de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" Jul 09 14:45:00.411068 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.410409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdb6\" (UniqueName: \"kubernetes.io/projected/6ac3dc7a-adba-4127-8023-917efc329045-kube-api-access-pzdb6\") pod \"maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p\" (UID: \"6ac3dc7a-adba-4127-8023-917efc329045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" Jul 09 14:45:00.470632 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.470596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" Jul 09 14:45:00.480367 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.480335 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" Jul 09 14:45:00.630180 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.630149 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq"] Jul 09 14:45:00.633337 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:45:00.633281 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07fa04a_8ab0_4ca5_b401_8ae31d2653de.slice/crio-0f4a3884795d6761946c508f58659d6be005c13cb32508393aa2e13de27e8cf5 WatchSource:0}: Error finding container 0f4a3884795d6761946c508f58659d6be005c13cb32508393aa2e13de27e8cf5: Status 404 returned error can't find the container with id 0f4a3884795d6761946c508f58659d6be005c13cb32508393aa2e13de27e8cf5 Jul 09 14:45:00.660309 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:00.660286 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p"] Jul 09 14:45:00.668494 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:45:00.668470 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac3dc7a_adba_4127_8023_917efc329045.slice/crio-1529e75cd56720ca7e0e6a020c7b53fb9c43ddd84367b74df2130cdfd4b8a2cb WatchSource:0}: Error finding container 1529e75cd56720ca7e0e6a020c7b53fb9c43ddd84367b74df2130cdfd4b8a2cb: Status 404 returned error can't find the container with id 1529e75cd56720ca7e0e6a020c7b53fb9c43ddd84367b74df2130cdfd4b8a2cb Jul 09 14:45:01.598405 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:01.598362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" event={"ID":"6ac3dc7a-adba-4127-8023-917efc329045","Type":"ContainerStarted","Data":"a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236"} Jul 09 14:45:01.598903 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:01.598462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" event={"ID":"6ac3dc7a-adba-4127-8023-917efc329045","Type":"ContainerStarted","Data":"1529e75cd56720ca7e0e6a020c7b53fb9c43ddd84367b74df2130cdfd4b8a2cb"} Jul 09 14:45:01.600092 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:01.600051 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" event={"ID":"e07fa04a-8ab0-4ca5-b401-8ae31d2653de","Type":"ContainerStarted","Data":"424da04c2e3dac4fc39b286cceb897175a086fcbb18c4c2783dd414b447db2d8"} Jul 09 14:45:01.600197 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:01.600122 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" event={"ID":"e07fa04a-8ab0-4ca5-b401-8ae31d2653de","Type":"ContainerStarted","Data":"0f4a3884795d6761946c508f58659d6be005c13cb32508393aa2e13de27e8cf5"} Jul 09 14:45:01.614847 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:01.614795 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" podStartSLOduration=1.6147777190000001 podStartE2EDuration="1.614777719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:01.613811828 +0000 UTC m=+1868.302741066" watchObservedRunningTime="2026-07-09 14:45:01.614777719 +0000 UTC m=+1868.303706957" Jul 09 14:45:01.629719 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:01.629659 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" podStartSLOduration=1.6296378599999999 podStartE2EDuration="1.62963786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:01.62909226 +0000 UTC m=+1868.318021499" watchObservedRunningTime="2026-07-09 14:45:01.62963786 +0000 UTC m=+1868.318567100" Jul 09 14:45:04.237767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.237734 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485"] Jul 09 14:45:04.238309 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.237944 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" podUID="40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" containerName="maas-api" containerID="cri-o://ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120" gracePeriod=30 Jul 09 14:45:04.284816 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.284774 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p"] Jul 09 14:45:04.285008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.284961 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" podUID="6ac3dc7a-adba-4127-8023-917efc329045" containerName="cleanup" containerID="cri-o://a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236" gracePeriod=30 Jul 09 14:45:04.502323 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.502255 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:45:04.618343 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.618304 2575 generic.go:356] "Generic (PLEG): container finished" podID="40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" containerID="ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120" exitCode=0 Jul 09 14:45:04.618555 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.618353 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" Jul 09 14:45:04.618555 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.618384 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" event={"ID":"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5","Type":"ContainerDied","Data":"ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120"} Jul 09 14:45:04.618555 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.618454 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485" event={"ID":"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5","Type":"ContainerDied","Data":"eb2e5f6068694522a262e1c2e90af4f8862c6097fef7d99e64dcb685b232c416"} Jul 09 14:45:04.618555 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.618485 2575 scope.go:117] "RemoveContainer" containerID="ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120" Jul 09 14:45:04.627628 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.627605 2575 scope.go:117] "RemoveContainer" containerID="ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120" Jul 09 14:45:04.627919 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:45:04.627883 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120\": container with ID starting with ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120 not found: ID does not exist" containerID="ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120" Jul 09 14:45:04.627987 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.627933 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120"} err="failed to get container status \"ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120\": rpc error: code = NotFound desc = could not find container \"ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120\": container with ID starting with ecbdf09efb8bc4ca4e077a1acac00674972ef1c9f7f69bcc1ec8fadfae8da120 not found: ID does not exist" Jul 09 14:45:04.639357 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.639326 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-openshift-service-ca\") pod \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " Jul 09 14:45:04.639511 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.639462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls\") pod \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " Jul 09 14:45:04.639511 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.639493 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx8dw\" (UniqueName: \"kubernetes.io/projected/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-kube-api-access-jx8dw\") pod \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\" (UID: \"40fed4cd-b8ee-4822-9f81-c7f2d299f5c5\") " Jul 09 14:45:04.639738 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.639710 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" (UID: "40fed4cd-b8ee-4822-9f81-c7f2d299f5c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:45:04.641573 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.641544 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" (UID: "40fed4cd-b8ee-4822-9f81-c7f2d299f5c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:45:04.641683 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.641602 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-kube-api-access-jx8dw" (OuterVolumeSpecName: "kube-api-access-jx8dw") pod "40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" (UID: "40fed4cd-b8ee-4822-9f81-c7f2d299f5c5"). InnerVolumeSpecName "kube-api-access-jx8dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:45:04.740785 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.740738 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:04.740785 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.740785 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8dw\" (UniqueName: \"kubernetes.io/projected/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-kube-api-access-jx8dw\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:04.740997 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.740800 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:04.948818 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.948783 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485"] Jul 09 14:45:04.953399 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:04.953369 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4777cd-d59fc4bc-8h485"] Jul 09 14:45:05.887739 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:05.887708 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" path="/var/lib/kubelet/pods/40fed4cd-b8ee-4822-9f81-c7f2d299f5c5/volumes" Jul 09 14:45:09.907308 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:09.907277 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs"] Jul 09 14:45:09.907777 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:09.907484 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" containerID="cri-o://fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c" gracePeriod=30 Jul 09 14:45:10.219959 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:10.219867 2575 patch_prober.go:28] interesting pod/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 14:45:10.219959 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:10.219934 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 14:45:15.175573 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.175501 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:45:15.239300 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239261 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-envoy\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239490 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239313 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-socket\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239490 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239359 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-credential-socket\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239490 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239398 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmr7h\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-kube-api-access-cmr7h\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239490 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239430 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-istio-token\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239490 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239461 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd51e30c-4060-4753-815e-620823a9fcdd-istio-podinfo\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239760 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239557 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:45:15.239760 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239629 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-certs\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239760 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239674 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-data\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.239760 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239691 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:45:15.239760 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd51e30c-4060-4753-815e-620823a9fcdd-istiod-ca-cert\") pod \"bd51e30c-4060-4753-815e-620823a9fcdd\" (UID: \"bd51e30c-4060-4753-815e-620823a9fcdd\") " Jul 09 14:45:15.240022 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.239890 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:45:15.240022 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.240017 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.240138 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.240031 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.240138 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.240041 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.240256 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.240161 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd51e30c-4060-4753-815e-620823a9fcdd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:45:15.240690 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.240659 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:45:15.241757 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.241727 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:45:15.241757 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.241787 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd51e30c-4060-4753-815e-620823a9fcdd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:45:15.241983 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.241907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-kube-api-access-cmr7h" (OuterVolumeSpecName: "kube-api-access-cmr7h") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "kube-api-access-cmr7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:45:15.242027 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.241968 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd51e30c-4060-4753-815e-620823a9fcdd" (UID: "bd51e30c-4060-4753-815e-620823a9fcdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:45:15.340603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.340552 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.340603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.340588 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmr7h\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-kube-api-access-cmr7h\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.340603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.340597 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd51e30c-4060-4753-815e-620823a9fcdd-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.340603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.340606 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd51e30c-4060-4753-815e-620823a9fcdd-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.340603 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.340613 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd51e30c-4060-4753-815e-620823a9fcdd-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.341013 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.340621 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd51e30c-4060-4753-815e-620823a9fcdd-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:15.681874 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.681839 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd51e30c-4060-4753-815e-620823a9fcdd" containerID="fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c" exitCode=0 Jul 09 14:45:15.682058 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.681891 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" Jul 09 14:45:15.682058 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.681928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" event={"ID":"bd51e30c-4060-4753-815e-620823a9fcdd","Type":"ContainerDied","Data":"fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c"} Jul 09 14:45:15.682058 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.681981 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs" event={"ID":"bd51e30c-4060-4753-815e-620823a9fcdd","Type":"ContainerDied","Data":"2fd95bbcac9b97ade90d0e55d0e863a55dbf172fc21c9c14f9e7203e44fcce54"} Jul 09 14:45:15.682058 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.682005 2575 scope.go:117] "RemoveContainer" containerID="fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c" Jul 09 14:45:15.693028 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.693009 2575 scope.go:117] "RemoveContainer" containerID="0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb" Jul 09 14:45:15.701784 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.701763 2575 scope.go:117] "RemoveContainer" containerID="fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c" Jul 09 14:45:15.702062 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:45:15.702017 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c\": container with ID starting with fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c not found: ID does not exist" containerID="fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c" Jul 09 14:45:15.702062 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.702043 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c"} err="failed to get container status \"fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c\": rpc error: code = NotFound desc = could not find container \"fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c\": container with ID starting with fc46271e1c12da1464144e458645b1263bf6ba68567cc7e067f4ad2c72b7483c not found: ID does not exist" Jul 09 14:45:15.702062 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.702059 2575 scope.go:117] "RemoveContainer" containerID="0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb" Jul 09 14:45:15.702282 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:45:15.702266 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb\": container with ID starting with 0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb not found: ID does not exist" containerID="0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb" Jul 09 14:45:15.702328 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.702286 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb"} err="failed to get container status \"0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb\": rpc error: code = NotFound desc = could not find container \"0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb\": container with ID starting with 0495e0257a167ab837ed5fcb47ccac72cf9f9c76a91d5d5aaf047a20ea76bbdb not found: ID does not exist" Jul 09 14:45:15.710292 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.710267 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs"] Jul 09 14:45:15.714838 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.714813 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4777cd-openshift-default-5995988c79-rs8xs"] Jul 09 14:45:15.888393 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:15.888360 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" path="/var/lib/kubelet/pods/bd51e30c-4060-4753-815e-620823a9fcdd/volumes" Jul 09 14:45:21.788132 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788085 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t"] Jul 09 14:45:21.788696 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788659 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" containerName="maas-api" Jul 09 14:45:21.788696 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788681 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" containerName="maas-api" Jul 09 14:45:21.788822 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788717 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" Jul 09 14:45:21.788822 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788727 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" Jul 09 14:45:21.788822 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788739 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" Jul 09 14:45:21.788822 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788748 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" Jul 09 14:45:21.788958 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788867 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" Jul 09 14:45:21.788958 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788897 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd51e30c-4060-4753-815e-620823a9fcdd" containerName="istio-proxy" Jul 09 14:45:21.788958 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.788908 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fed4cd-b8ee-4822-9f81-c7f2d299f5c5" containerName="maas-api" Jul 09 14:45:21.793610 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.793590 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.796336 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.796313 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-41b03c-serving-cert\"" Jul 09 14:45:21.802976 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.802949 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t"] Jul 09 14:45:21.892154 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.892116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56mr\" (UniqueName: \"kubernetes.io/projected/5f82f76a-4547-4c05-819e-28cdf5907e22-kube-api-access-q56mr\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.892154 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.892160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f82f76a-4547-4c05-819e-28cdf5907e22-maas-api-tls\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.892439 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.892302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f82f76a-4547-4c05-819e-28cdf5907e22-openshift-service-ca\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.992983 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.992930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56mr\" (UniqueName: \"kubernetes.io/projected/5f82f76a-4547-4c05-819e-28cdf5907e22-kube-api-access-q56mr\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.992983 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.992991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f82f76a-4547-4c05-819e-28cdf5907e22-maas-api-tls\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.993211 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.993047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f82f76a-4547-4c05-819e-28cdf5907e22-openshift-service-ca\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.993741 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.993718 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f82f76a-4547-4c05-819e-28cdf5907e22-openshift-service-ca\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:21.995819 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:21.995788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f82f76a-4547-4c05-819e-28cdf5907e22-maas-api-tls\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:22.002299 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:22.002274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56mr\" (UniqueName: \"kubernetes.io/projected/5f82f76a-4547-4c05-819e-28cdf5907e22-kube-api-access-q56mr\") pod \"maas-api-e2e-api-b-41b03c-64957c7d89-gt22t\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:22.106788 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:22.106690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:22.252953 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:22.252917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t"] Jul 09 14:45:22.254832 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:45:22.254806 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f82f76a_4547_4c05_819e_28cdf5907e22.slice/crio-35ecdbfab5bdc543504c692d8cb9658c179435347d0fa3989ba5d2b1f28cce00 WatchSource:0}: Error finding container 35ecdbfab5bdc543504c692d8cb9658c179435347d0fa3989ba5d2b1f28cce00: Status 404 returned error can't find the container with id 35ecdbfab5bdc543504c692d8cb9658c179435347d0fa3989ba5d2b1f28cce00 Jul 09 14:45:22.719491 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:22.719453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" event={"ID":"5f82f76a-4547-4c05-819e-28cdf5907e22","Type":"ContainerStarted","Data":"35ecdbfab5bdc543504c692d8cb9658c179435347d0fa3989ba5d2b1f28cce00"} Jul 09 14:45:23.726659 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:23.726622 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" event={"ID":"5f82f76a-4547-4c05-819e-28cdf5907e22","Type":"ContainerStarted","Data":"95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb"} Jul 09 14:45:23.727060 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:23.726704 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:23.744127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:23.744069 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" podStartSLOduration=2.208172296 podStartE2EDuration="2.744049382s" totalImagesPullingTime="535.877086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:23.742721832 +0000 UTC m=+1890.431651073" watchObservedRunningTime="2026-07-09 14:45:23.744049382 +0000 UTC m=+1890.432978675" Jul 09 14:45:29.737331 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:29.737297 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:34.484449 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.484416 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" Jul 09 14:45:34.604113 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.604016 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzdb6\" (UniqueName: \"kubernetes.io/projected/6ac3dc7a-adba-4127-8023-917efc329045-kube-api-access-pzdb6\") pod \"6ac3dc7a-adba-4127-8023-917efc329045\" (UID: \"6ac3dc7a-adba-4127-8023-917efc329045\") " Jul 09 14:45:34.606113 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.606079 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac3dc7a-adba-4127-8023-917efc329045-kube-api-access-pzdb6" (OuterVolumeSpecName: "kube-api-access-pzdb6") pod "6ac3dc7a-adba-4127-8023-917efc329045" (UID: "6ac3dc7a-adba-4127-8023-917efc329045"). InnerVolumeSpecName "kube-api-access-pzdb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:45:34.705696 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.705658 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzdb6\" (UniqueName: \"kubernetes.io/projected/6ac3dc7a-adba-4127-8023-917efc329045-kube-api-access-pzdb6\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:34.786768 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.786729 2575 generic.go:356] "Generic (PLEG): container finished" podID="6ac3dc7a-adba-4127-8023-917efc329045" containerID="a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236" exitCode=137 Jul 09 14:45:34.786982 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.786776 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" Jul 09 14:45:34.786982 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.786807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" event={"ID":"6ac3dc7a-adba-4127-8023-917efc329045","Type":"ContainerDied","Data":"a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236"} Jul 09 14:45:34.786982 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.786858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p" event={"ID":"6ac3dc7a-adba-4127-8023-917efc329045","Type":"ContainerDied","Data":"1529e75cd56720ca7e0e6a020c7b53fb9c43ddd84367b74df2130cdfd4b8a2cb"} Jul 09 14:45:34.786982 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.786881 2575 scope.go:117] "RemoveContainer" containerID="a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236" Jul 09 14:45:34.796330 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.796307 2575 scope.go:117] "RemoveContainer" containerID="a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236" Jul 09 14:45:34.796600 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:45:34.796580 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236\": container with ID starting with a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236 not found: ID does not exist" containerID="a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236" Jul 09 14:45:34.796651 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.796610 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236"} err="failed to get container status \"a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236\": rpc error: code = NotFound desc = could not find container \"a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236\": container with ID starting with a288f520b1e8a933e84e6bc904b5fec5bfbb0bb15cbf2b992acf0948df98e236 not found: ID does not exist" Jul 09 14:45:34.810406 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.810380 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p"] Jul 09 14:45:34.817139 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:34.817110 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4777cd-29726805-nsv6p"] Jul 09 14:45:35.885846 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:35.885810 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac3dc7a-adba-4127-8023-917efc329045" path="/var/lib/kubelet/pods/6ac3dc7a-adba-4127-8023-917efc329045/volumes" Jul 09 14:45:42.451035 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.450998 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t"] Jul 09 14:45:42.451570 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.451306 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" podUID="5f82f76a-4547-4c05-819e-28cdf5907e22" containerName="maas-api" containerID="cri-o://95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb" gracePeriod=30 Jul 09 14:45:42.722800 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.722774 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:42.777178 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.777145 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q56mr\" (UniqueName: \"kubernetes.io/projected/5f82f76a-4547-4c05-819e-28cdf5907e22-kube-api-access-q56mr\") pod \"5f82f76a-4547-4c05-819e-28cdf5907e22\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " Jul 09 14:45:42.777375 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.777202 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f82f76a-4547-4c05-819e-28cdf5907e22-openshift-service-ca\") pod \"5f82f76a-4547-4c05-819e-28cdf5907e22\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " Jul 09 14:45:42.777375 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.777249 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f82f76a-4547-4c05-819e-28cdf5907e22-maas-api-tls\") pod \"5f82f76a-4547-4c05-819e-28cdf5907e22\" (UID: \"5f82f76a-4547-4c05-819e-28cdf5907e22\") " Jul 09 14:45:42.777571 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.777540 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f82f76a-4547-4c05-819e-28cdf5907e22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f82f76a-4547-4c05-819e-28cdf5907e22" (UID: "5f82f76a-4547-4c05-819e-28cdf5907e22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:45:42.779254 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.779217 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f82f76a-4547-4c05-819e-28cdf5907e22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f82f76a-4547-4c05-819e-28cdf5907e22" (UID: "5f82f76a-4547-4c05-819e-28cdf5907e22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:45:42.779361 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.779339 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f82f76a-4547-4c05-819e-28cdf5907e22-kube-api-access-q56mr" (OuterVolumeSpecName: "kube-api-access-q56mr") pod "5f82f76a-4547-4c05-819e-28cdf5907e22" (UID: "5f82f76a-4547-4c05-819e-28cdf5907e22"). InnerVolumeSpecName "kube-api-access-q56mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:45:42.829354 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.829323 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f82f76a-4547-4c05-819e-28cdf5907e22" containerID="95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb" exitCode=0 Jul 09 14:45:42.829544 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.829371 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" Jul 09 14:45:42.829544 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.829408 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" event={"ID":"5f82f76a-4547-4c05-819e-28cdf5907e22","Type":"ContainerDied","Data":"95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb"} Jul 09 14:45:42.829544 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.829467 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t" event={"ID":"5f82f76a-4547-4c05-819e-28cdf5907e22","Type":"ContainerDied","Data":"35ecdbfab5bdc543504c692d8cb9658c179435347d0fa3989ba5d2b1f28cce00"} Jul 09 14:45:42.829544 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.829499 2575 scope.go:117] "RemoveContainer" containerID="95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb" Jul 09 14:45:42.839516 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.839495 2575 scope.go:117] "RemoveContainer" containerID="95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb" Jul 09 14:45:42.839816 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:45:42.839796 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb\": container with ID starting with 95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb not found: ID does not exist" containerID="95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb" Jul 09 14:45:42.839862 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.839829 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb"} err="failed to get container status \"95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb\": rpc error: code = NotFound desc = could not find container \"95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb\": container with ID starting with 95c218f4090385ae86b8a7a4b1ace8f43c64f92db83e04ec4b388751871357cb not found: ID does not exist" Jul 09 14:45:42.851722 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.851697 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t"] Jul 09 14:45:42.855788 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.855767 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41b03c-64957c7d89-gt22t"] Jul 09 14:45:42.878100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.878068 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56mr\" (UniqueName: \"kubernetes.io/projected/5f82f76a-4547-4c05-819e-28cdf5907e22-kube-api-access-q56mr\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:42.878100 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.878096 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f82f76a-4547-4c05-819e-28cdf5907e22-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:42.878318 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:42.878108 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f82f76a-4547-4c05-819e-28cdf5907e22-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:45:43.890831 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:43.890795 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f82f76a-4547-4c05-819e-28cdf5907e22" path="/var/lib/kubelet/pods/5f82f76a-4547-4c05-819e-28cdf5907e22/volumes" Jul 09 14:45:49.374727 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.374694 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s"] Jul 09 14:45:49.375213 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.375133 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac3dc7a-adba-4127-8023-917efc329045" containerName="cleanup" Jul 09 14:45:49.375213 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.375144 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac3dc7a-adba-4127-8023-917efc329045" containerName="cleanup" Jul 09 14:45:49.375213 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.375166 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f82f76a-4547-4c05-819e-28cdf5907e22" containerName="maas-api" Jul 09 14:45:49.375213 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.375172 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f82f76a-4547-4c05-819e-28cdf5907e22" containerName="maas-api" Jul 09 14:45:49.375451 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.375254 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac3dc7a-adba-4127-8023-917efc329045" containerName="cleanup" Jul 09 14:45:49.375451 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.375265 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f82f76a-4547-4c05-819e-28cdf5907e22" containerName="maas-api" Jul 09 14:45:49.379548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.379528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.383420 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.383399 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b22ad3-openshift-default-dockercfg-gww5c\"" Jul 09 14:45:49.383645 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.383632 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 14:45:49.396430 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.396399 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s"] Jul 09 14:45:49.437366 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istiod-ca-cert\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-podinfo\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-envoy\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-token\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnztp\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-kube-api-access-mnztp\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437729 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-socket\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437729 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-data\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437834 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-credential-socket\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.437834 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.437783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-certs\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.538804 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.538763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istiod-ca-cert\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539026 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.538820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-podinfo\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539026 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.538879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-envoy\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539026 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.538916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-token\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539026 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.538971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnztp\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-kube-api-access-mnztp\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539304 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-socket\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539304 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-data\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539304 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-credential-socket\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539304 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-certs\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539527 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-socket\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539527 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-certs\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-data\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539742 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-credential-socket\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.539742 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.539653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istiod-ca-cert\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.541319 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.541300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-envoy\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.541536 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.541517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-podinfo\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.548309 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.548287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-token\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.548644 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.548624 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnztp\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-kube-api-access-mnztp\") pod \"e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.689874 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.689783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:49.836461 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.836431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s"] Jul 09 14:45:49.838569 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:45:49.838543 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc8eb6a_b410_4522_8c07_03dde0e4e028.slice/crio-1dad508191dd4207b7be229d96787f5f980c3d09017c04b8886699e8246c0a00 WatchSource:0}: Error finding container 1dad508191dd4207b7be229d96787f5f980c3d09017c04b8886699e8246c0a00: Status 404 returned error can't find the container with id 1dad508191dd4207b7be229d96787f5f980c3d09017c04b8886699e8246c0a00 Jul 09 14:45:49.841032 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.840997 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:45:49.841120 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.841066 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:45:49.841120 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.841099 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:45:49.869712 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:49.869676 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" event={"ID":"fdc8eb6a-b410-4522-8c07-03dde0e4e028","Type":"ContainerStarted","Data":"1dad508191dd4207b7be229d96787f5f980c3d09017c04b8886699e8246c0a00"} Jul 09 14:45:50.876170 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:50.876135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" event={"ID":"fdc8eb6a-b410-4522-8c07-03dde0e4e028","Type":"ContainerStarted","Data":"d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd"} Jul 09 14:45:50.896573 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:50.896524 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" podStartSLOduration=1.896504124 podStartE2EDuration="1.896504124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:50.896125947 +0000 UTC m=+1917.585055186" watchObservedRunningTime="2026-07-09 14:45:50.896504124 +0000 UTC m=+1917.585433364" Jul 09 14:45:51.690376 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:51.690340 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:51.695857 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:51.695831 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:51.886830 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:51.886799 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:45:51.887214 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:45:51.886875 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:46:00.013122 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:00.013084 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq"] Jul 09 14:46:00.013611 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:00.013296 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" podUID="e07fa04a-8ab0-4ca5-b401-8ae31d2653de" containerName="cleanup" containerID="cri-o://424da04c2e3dac4fc39b286cceb897175a086fcbb18c4c2783dd414b447db2d8" gracePeriod=30 Jul 09 14:46:00.876700 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:00.876662 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt"] Jul 09 14:46:00.880300 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:00.880276 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:00.882550 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:00.882522 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c4eda7-serving-cert\"" Jul 09 14:46:00.892901 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:00.892878 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt"] Jul 09 14:46:01.052949 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.052902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.053720 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.053690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.054335 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.054315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhsq\" (UniqueName: \"kubernetes.io/projected/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-kube-api-access-qrhsq\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.156510 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.156407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.156696 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.156511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhsq\" (UniqueName: \"kubernetes.io/projected/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-kube-api-access-qrhsq\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.157112 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.157084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.157713 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.157689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.160811 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.160788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-maas-api-tls\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.166183 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.166135 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhsq\" (UniqueName: \"kubernetes.io/projected/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-kube-api-access-qrhsq\") pod \"maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.193106 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.193071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.381350 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.381322 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt"] Jul 09 14:46:01.384157 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:46:01.384133 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f0c3e6_66b7_41e7_8f52_97a94f7b1b55.slice/crio-42340e62d71026cdb3d750fc591842a9194718e462b4afd6af1026d8fe7ccd7f WatchSource:0}: Error finding container 42340e62d71026cdb3d750fc591842a9194718e462b4afd6af1026d8fe7ccd7f: Status 404 returned error can't find the container with id 42340e62d71026cdb3d750fc591842a9194718e462b4afd6af1026d8fe7ccd7f Jul 09 14:46:01.939120 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.939075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" event={"ID":"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55","Type":"ContainerStarted","Data":"82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a"} Jul 09 14:46:01.939348 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.939147 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" event={"ID":"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55","Type":"ContainerStarted","Data":"42340e62d71026cdb3d750fc591842a9194718e462b4afd6af1026d8fe7ccd7f"} Jul 09 14:46:01.940030 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.939998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:01.958265 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:01.958177 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" podStartSLOduration=1.490687413 podStartE2EDuration="1.958153685s" totalImagesPullingTime="467.466272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:46:01.954557836 +0000 UTC m=+1928.643487096" watchObservedRunningTime="2026-07-09 14:46:01.958153685 +0000 UTC m=+1928.647082925" Jul 09 14:46:04.957850 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:04.957816 2575 generic.go:356] "Generic (PLEG): container finished" podID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerID="d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd" exitCode=0 Jul 09 14:46:04.958394 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:04.957897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" event={"ID":"fdc8eb6a-b410-4522-8c07-03dde0e4e028","Type":"ContainerDied","Data":"d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd"} Jul 09 14:46:04.958472 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:04.958409 2575 scope.go:117] "RemoveContainer" containerID="d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd" Jul 09 14:46:04.959496 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:04.959469 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:46:04.959573 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:04.959539 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:46:04.959573 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:04.959569 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 14:46:05.966959 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:05.966923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" event={"ID":"fdc8eb6a-b410-4522-8c07-03dde0e4e028","Type":"ContainerStarted","Data":"8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7"} Jul 09 14:46:06.690601 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:06.690565 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:46:06.691962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:06.691932 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:46:06.692089 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:06.691985 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:46:06.880581 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:06.880541 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:46:07.691000 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:07.690965 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:46:07.691442 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:07.691024 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:46:08.690233 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:08.690198 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:46:08.690415 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:08.690296 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:46:08.963388 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:08.963292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:46:09.690700 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:09.690660 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:46:09.690887 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:09.690736 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:46:10.695338 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:10.695304 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:46:10.696327 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:10.696306 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:46:30.096636 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:30.096603 2575 generic.go:356] "Generic (PLEG): container finished" podID="e07fa04a-8ab0-4ca5-b401-8ae31d2653de" containerID="424da04c2e3dac4fc39b286cceb897175a086fcbb18c4c2783dd414b447db2d8" exitCode=137 Jul 09 14:46:30.097149 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:30.096650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" event={"ID":"e07fa04a-8ab0-4ca5-b401-8ae31d2653de","Type":"ContainerDied","Data":"424da04c2e3dac4fc39b286cceb897175a086fcbb18c4c2783dd414b447db2d8"} Jul 09 14:46:30.201082 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:30.201052 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" Jul 09 14:46:30.334713 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:30.334672 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld5bq\" (UniqueName: \"kubernetes.io/projected/e07fa04a-8ab0-4ca5-b401-8ae31d2653de-kube-api-access-ld5bq\") pod \"e07fa04a-8ab0-4ca5-b401-8ae31d2653de\" (UID: \"e07fa04a-8ab0-4ca5-b401-8ae31d2653de\") " Jul 09 14:46:30.336932 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:30.336898 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07fa04a-8ab0-4ca5-b401-8ae31d2653de-kube-api-access-ld5bq" (OuterVolumeSpecName: "kube-api-access-ld5bq") pod "e07fa04a-8ab0-4ca5-b401-8ae31d2653de" (UID: "e07fa04a-8ab0-4ca5-b401-8ae31d2653de"). InnerVolumeSpecName "kube-api-access-ld5bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:46:30.435661 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:30.435614 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5bq\" (UniqueName: \"kubernetes.io/projected/e07fa04a-8ab0-4ca5-b401-8ae31d2653de-kube-api-access-ld5bq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:46:31.103168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:31.103138 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" Jul 09 14:46:31.103168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:31.103156 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq" event={"ID":"e07fa04a-8ab0-4ca5-b401-8ae31d2653de","Type":"ContainerDied","Data":"0f4a3884795d6761946c508f58659d6be005c13cb32508393aa2e13de27e8cf5"} Jul 09 14:46:31.103676 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:31.103233 2575 scope.go:117] "RemoveContainer" containerID="424da04c2e3dac4fc39b286cceb897175a086fcbb18c4c2783dd414b447db2d8" Jul 09 14:46:31.133288 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:31.133260 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq"] Jul 09 14:46:31.137108 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:31.137084 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-t9rtq"] Jul 09 14:46:31.886886 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:46:31.886852 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07fa04a-8ab0-4ca5-b401-8ae31d2653de" path="/var/lib/kubelet/pods/e07fa04a-8ab0-4ca5-b401-8ae31d2653de/volumes" Jul 09 14:48:53.960756 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:48:53.960725 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:48:53.969765 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:48:53.969740 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:51:54.835762 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.835721 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7566-140598039660080"] Jul 09 14:51:54.836393 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.836374 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07fa04a-8ab0-4ca5-b401-8ae31d2653de" containerName="cleanup" Jul 09 14:51:54.836460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.836397 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07fa04a-8ab0-4ca5-b401-8ae31d2653de" containerName="cleanup" Jul 09 14:51:54.836526 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.836514 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07fa04a-8ab0-4ca5-b401-8ae31d2653de" containerName="cleanup" Jul 09 14:51:54.839985 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.839965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:54.842582 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.842560 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-k57nw\"" Jul 09 14:51:54.845538 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.845516 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598039660080"] Jul 09 14:51:54.930833 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:54.930795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69rp\" (UniqueName: \"kubernetes.io/projected/0e844ffb-9afc-421c-9b35-e99b6d76281b-kube-api-access-j69rp\") pod \"test-curl-7566-140598039660080\" (UID: \"0e844ffb-9afc-421c-9b35-e99b6d76281b\") " pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:55.031688 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:55.031655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69rp\" (UniqueName: \"kubernetes.io/projected/0e844ffb-9afc-421c-9b35-e99b6d76281b-kube-api-access-j69rp\") pod \"test-curl-7566-140598039660080\" (UID: \"0e844ffb-9afc-421c-9b35-e99b6d76281b\") " pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:55.039958 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:55.039922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69rp\" (UniqueName: \"kubernetes.io/projected/0e844ffb-9afc-421c-9b35-e99b6d76281b-kube-api-access-j69rp\") pod \"test-curl-7566-140598039660080\" (UID: \"0e844ffb-9afc-421c-9b35-e99b6d76281b\") " pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:55.150379 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:55.150287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:55.293698 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:55.293667 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598039660080"] Jul 09 14:51:55.296238 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:51:55.296198 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e844ffb_9afc_421c_9b35_e99b6d76281b.slice/crio-ceaa6317d022105be7a1ffbd8389cd3657eb978b21f6e7eb4407c43d08fc4d3d WatchSource:0}: Error finding container ceaa6317d022105be7a1ffbd8389cd3657eb978b21f6e7eb4407c43d08fc4d3d: Status 404 returned error can't find the container with id ceaa6317d022105be7a1ffbd8389cd3657eb978b21f6e7eb4407c43d08fc4d3d Jul 09 14:51:55.297619 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:55.297598 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:51:55.782443 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:55.782394 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598039660080" event={"ID":"0e844ffb-9afc-421c-9b35-e99b6d76281b","Type":"ContainerStarted","Data":"ceaa6317d022105be7a1ffbd8389cd3657eb978b21f6e7eb4407c43d08fc4d3d"} Jul 09 14:51:56.788678 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:56.788645 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e844ffb-9afc-421c-9b35-e99b6d76281b" containerID="a7da0c92eb9c2444fae66715c5628956a155074852f34d51fce2c46caad868a0" exitCode=0 Jul 09 14:51:56.789092 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:56.788728 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598039660080" event={"ID":"0e844ffb-9afc-421c-9b35-e99b6d76281b","Type":"ContainerDied","Data":"a7da0c92eb9c2444fae66715c5628956a155074852f34d51fce2c46caad868a0"} Jul 09 14:51:57.944997 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:57.944970 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:57.987615 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:57.987589 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7566-140598039660080_0e844ffb-9afc-421c-9b35-e99b6d76281b/test-curl-7566-140598039660080/0.log" Jul 09 14:51:58.010917 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.010876 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7566-140598039660080"] Jul 09 14:51:58.018464 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.018441 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7566-140598039660080"] Jul 09 14:51:58.061341 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.061251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j69rp\" (UniqueName: \"kubernetes.io/projected/0e844ffb-9afc-421c-9b35-e99b6d76281b-kube-api-access-j69rp\") pod \"0e844ffb-9afc-421c-9b35-e99b6d76281b\" (UID: \"0e844ffb-9afc-421c-9b35-e99b6d76281b\") " Jul 09 14:51:58.063385 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.063358 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e844ffb-9afc-421c-9b35-e99b6d76281b-kube-api-access-j69rp" (OuterVolumeSpecName: "kube-api-access-j69rp") pod "0e844ffb-9afc-421c-9b35-e99b6d76281b" (UID: "0e844ffb-9afc-421c-9b35-e99b6d76281b"). InnerVolumeSpecName "kube-api-access-j69rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:51:58.149161 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.149129 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7566-140598038294704"] Jul 09 14:51:58.149563 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.149550 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e844ffb-9afc-421c-9b35-e99b6d76281b" containerName="test-curl-7566-140598039660080" Jul 09 14:51:58.149614 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.149565 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e844ffb-9afc-421c-9b35-e99b6d76281b" containerName="test-curl-7566-140598039660080" Jul 09 14:51:58.149653 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.149633 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e844ffb-9afc-421c-9b35-e99b6d76281b" containerName="test-curl-7566-140598039660080" Jul 09 14:51:58.152668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.152653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:51:58.162137 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.162113 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69rp\" (UniqueName: \"kubernetes.io/projected/0e844ffb-9afc-421c-9b35-e99b6d76281b-kube-api-access-j69rp\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.171025 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.170997 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598038294704"] Jul 09 14:51:58.263166 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.263119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdk2\" (UniqueName: \"kubernetes.io/projected/6d21942c-2bb9-417d-8c32-86cf7595cb40-kube-api-access-prdk2\") pod \"test-curl-7566-140598038294704\" (UID: \"6d21942c-2bb9-417d-8c32-86cf7595cb40\") " pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:51:58.363965 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.363867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdk2\" (UniqueName: \"kubernetes.io/projected/6d21942c-2bb9-417d-8c32-86cf7595cb40-kube-api-access-prdk2\") pod \"test-curl-7566-140598038294704\" (UID: \"6d21942c-2bb9-417d-8c32-86cf7595cb40\") " pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:51:58.374306 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.374276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdk2\" (UniqueName: \"kubernetes.io/projected/6d21942c-2bb9-417d-8c32-86cf7595cb40-kube-api-access-prdk2\") pod \"test-curl-7566-140598038294704\" (UID: \"6d21942c-2bb9-417d-8c32-86cf7595cb40\") " pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:51:58.461117 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.461081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:51:58.612700 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.612672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598038294704"] Jul 09 14:51:58.614215 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:51:58.614158 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d21942c_2bb9_417d_8c32_86cf7595cb40.slice/crio-0c07099e19eb6fe9dde17fc59ab25676a01de646818da018b35667a7dbe0490b WatchSource:0}: Error finding container 0c07099e19eb6fe9dde17fc59ab25676a01de646818da018b35667a7dbe0490b: Status 404 returned error can't find the container with id 0c07099e19eb6fe9dde17fc59ab25676a01de646818da018b35667a7dbe0490b Jul 09 14:51:58.800174 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.800147 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:58.800174 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.800173 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceaa6317d022105be7a1ffbd8389cd3657eb978b21f6e7eb4407c43d08fc4d3d" Jul 09 14:51:58.801369 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:58.801341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598038294704" event={"ID":"6d21942c-2bb9-417d-8c32-86cf7595cb40","Type":"ContainerStarted","Data":"0c07099e19eb6fe9dde17fc59ab25676a01de646818da018b35667a7dbe0490b"} Jul 09 14:51:58.810920 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:51:58.810886 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7566-140598039660080\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0e844ffb-9afc-421c-9b35-e99b6d76281b" pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:59.807628 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:59.807585 2575 generic.go:356] "Generic (PLEG): container finished" podID="6d21942c-2bb9-417d-8c32-86cf7595cb40" containerID="d11188c75e965770f3711c0f61fc41fa3c890a509ee0e670267ee0db8d4cd2a2" exitCode=0 Jul 09 14:51:59.808025 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:59.807647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598038294704" event={"ID":"6d21942c-2bb9-417d-8c32-86cf7595cb40","Type":"ContainerDied","Data":"d11188c75e965770f3711c0f61fc41fa3c890a509ee0e670267ee0db8d4cd2a2"} Jul 09 14:51:59.825625 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:51:59.825586 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7566-140598039660080\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="0e844ffb-9afc-421c-9b35-e99b6d76281b" pod="openshift-ingress/test-curl-7566-140598039660080" Jul 09 14:51:59.887180 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:51:59.887144 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e844ffb-9afc-421c-9b35-e99b6d76281b" path="/var/lib/kubelet/pods/0e844ffb-9afc-421c-9b35-e99b6d76281b/volumes" Jul 09 14:52:00.958078 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:00.958054 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:52:00.979323 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:00.979276 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7566-140598038294704_6d21942c-2bb9-417d-8c32-86cf7595cb40/test-curl-7566-140598038294704/0.log" Jul 09 14:52:01.001445 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.001414 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7566-140598038294704"] Jul 09 14:52:01.006498 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.006475 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7566-140598038294704"] Jul 09 14:52:01.087882 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.087794 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prdk2\" (UniqueName: \"kubernetes.io/projected/6d21942c-2bb9-417d-8c32-86cf7595cb40-kube-api-access-prdk2\") pod \"6d21942c-2bb9-417d-8c32-86cf7595cb40\" (UID: \"6d21942c-2bb9-417d-8c32-86cf7595cb40\") " Jul 09 14:52:01.089883 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.089854 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d21942c-2bb9-417d-8c32-86cf7595cb40-kube-api-access-prdk2" (OuterVolumeSpecName: "kube-api-access-prdk2") pod "6d21942c-2bb9-417d-8c32-86cf7595cb40" (UID: "6d21942c-2bb9-417d-8c32-86cf7595cb40"). InnerVolumeSpecName "kube-api-access-prdk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:01.159492 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.159461 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:01.159883 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.159869 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d21942c-2bb9-417d-8c32-86cf7595cb40" containerName="test-curl-7566-140598038294704" Jul 09 14:52:01.159928 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.159885 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d21942c-2bb9-417d-8c32-86cf7595cb40" containerName="test-curl-7566-140598038294704" Jul 09 14:52:01.159964 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.159944 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d21942c-2bb9-417d-8c32-86cf7595cb40" containerName="test-curl-7566-140598038294704" Jul 09 14:52:01.162990 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.162974 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:01.170701 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.170672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:01.189593 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.189564 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdk2\" (UniqueName: \"kubernetes.io/projected/6d21942c-2bb9-417d-8c32-86cf7595cb40-kube-api-access-prdk2\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:01.290433 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.290393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lg6x\" (UniqueName: \"kubernetes.io/projected/f3a62e69-5efb-4879-851a-35602807bc84-kube-api-access-8lg6x\") pod \"test-curl-7566-140598040319296\" (UID: \"f3a62e69-5efb-4879-851a-35602807bc84\") " pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:01.391875 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.391769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lg6x\" (UniqueName: \"kubernetes.io/projected/f3a62e69-5efb-4879-851a-35602807bc84-kube-api-access-8lg6x\") pod \"test-curl-7566-140598040319296\" (UID: \"f3a62e69-5efb-4879-851a-35602807bc84\") " pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:01.400825 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.400792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lg6x\" (UniqueName: \"kubernetes.io/projected/f3a62e69-5efb-4879-851a-35602807bc84-kube-api-access-8lg6x\") pod \"test-curl-7566-140598040319296\" (UID: \"f3a62e69-5efb-4879-851a-35602807bc84\") " pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:01.473067 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.473029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:01.620753 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.620725 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:01.622490 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:52:01.622461 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3a62e69_5efb_4879_851a_35602807bc84.slice/crio-4230bd7f75ade429766a2af07afee81a46415b551d95680d768ec2e7f30783f2 WatchSource:0}: Error finding container 4230bd7f75ade429766a2af07afee81a46415b551d95680d768ec2e7f30783f2: Status 404 returned error can't find the container with id 4230bd7f75ade429766a2af07afee81a46415b551d95680d768ec2e7f30783f2 Jul 09 14:52:01.820040 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.819999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040319296" event={"ID":"f3a62e69-5efb-4879-851a-35602807bc84","Type":"ContainerStarted","Data":"4230bd7f75ade429766a2af07afee81a46415b551d95680d768ec2e7f30783f2"} Jul 09 14:52:01.821261 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.821218 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c07099e19eb6fe9dde17fc59ab25676a01de646818da018b35667a7dbe0490b" Jul 09 14:52:01.821366 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.821260 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:52:01.832132 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:52:01.832101 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7566-140598038294704\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="6d21942c-2bb9-417d-8c32-86cf7595cb40" pod="openshift-ingress/test-curl-7566-140598038294704" Jul 09 14:52:01.886413 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:01.886383 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d21942c-2bb9-417d-8c32-86cf7595cb40" path="/var/lib/kubelet/pods/6d21942c-2bb9-417d-8c32-86cf7595cb40/volumes" Jul 09 14:52:02.828320 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:02.828289 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3a62e69-5efb-4879-851a-35602807bc84" containerID="2f78431faac3e1e09e626605cb2418ef5aadac31eff0f581c1f50aad590ce615" exitCode=0 Jul 09 14:52:02.828721 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:02.828377 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040319296" event={"ID":"f3a62e69-5efb-4879-851a-35602807bc84","Type":"ContainerDied","Data":"2f78431faac3e1e09e626605cb2418ef5aadac31eff0f581c1f50aad590ce615"} Jul 09 14:52:03.993694 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:03.993667 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:04.025783 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.025759 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7566-140598040319296_f3a62e69-5efb-4879-851a-35602807bc84/test-curl-7566-140598040319296/0.log" Jul 09 14:52:04.052127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.052083 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:04.057992 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.057966 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:04.118985 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.118895 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lg6x\" (UniqueName: \"kubernetes.io/projected/f3a62e69-5efb-4879-851a-35602807bc84-kube-api-access-8lg6x\") pod \"f3a62e69-5efb-4879-851a-35602807bc84\" (UID: \"f3a62e69-5efb-4879-851a-35602807bc84\") " Jul 09 14:52:04.121005 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.120971 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a62e69-5efb-4879-851a-35602807bc84-kube-api-access-8lg6x" (OuterVolumeSpecName: "kube-api-access-8lg6x") pod "f3a62e69-5efb-4879-851a-35602807bc84" (UID: "f3a62e69-5efb-4879-851a-35602807bc84"). InnerVolumeSpecName "kube-api-access-8lg6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:04.184179 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.184144 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:04.184593 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.184578 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a62e69-5efb-4879-851a-35602807bc84" containerName="test-curl-7566-140598040319296" Jul 09 14:52:04.184639 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.184596 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a62e69-5efb-4879-851a-35602807bc84" containerName="test-curl-7566-140598040319296" Jul 09 14:52:04.184679 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.184674 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a62e69-5efb-4879-851a-35602807bc84" containerName="test-curl-7566-140598040319296" Jul 09 14:52:04.187703 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.187686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:04.196259 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.196212 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:04.220562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.220530 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lg6x\" (UniqueName: \"kubernetes.io/projected/f3a62e69-5efb-4879-851a-35602807bc84-kube-api-access-8lg6x\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:04.321683 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.321638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fds\" (UniqueName: \"kubernetes.io/projected/28e88da1-b991-43f8-939f-b6104f678d9f-kube-api-access-r7fds\") pod \"test-curl-7566-140598040319296\" (UID: \"28e88da1-b991-43f8-939f-b6104f678d9f\") " pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:04.422803 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.422712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fds\" (UniqueName: \"kubernetes.io/projected/28e88da1-b991-43f8-939f-b6104f678d9f-kube-api-access-r7fds\") pod \"test-curl-7566-140598040319296\" (UID: \"28e88da1-b991-43f8-939f-b6104f678d9f\") " pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:04.432744 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.432711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fds\" (UniqueName: \"kubernetes.io/projected/28e88da1-b991-43f8-939f-b6104f678d9f-kube-api-access-r7fds\") pod \"test-curl-7566-140598040319296\" (UID: \"28e88da1-b991-43f8-939f-b6104f678d9f\") " pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:04.497474 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.497439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:04.652720 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.652583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:04.654989 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:52:04.654959 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28e88da1_b991_43f8_939f_b6104f678d9f.slice/crio-20075af0febc08b6175a91586ba97a9167d5a11bc191b8ebf91cfc6e082a3063 WatchSource:0}: Error finding container 20075af0febc08b6175a91586ba97a9167d5a11bc191b8ebf91cfc6e082a3063: Status 404 returned error can't find the container with id 20075af0febc08b6175a91586ba97a9167d5a11bc191b8ebf91cfc6e082a3063 Jul 09 14:52:04.840640 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.840608 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:04.840801 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.840608 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4230bd7f75ade429766a2af07afee81a46415b551d95680d768ec2e7f30783f2" Jul 09 14:52:04.841826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.841799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040319296" event={"ID":"28e88da1-b991-43f8-939f-b6104f678d9f","Type":"ContainerStarted","Data":"20075af0febc08b6175a91586ba97a9167d5a11bc191b8ebf91cfc6e082a3063"} Jul 09 14:52:04.856027 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:04.855987 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7566-140598040319296" oldPodUID="f3a62e69-5efb-4879-851a-35602807bc84" podUID="28e88da1-b991-43f8-939f-b6104f678d9f" Jul 09 14:52:05.849133 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:05.849101 2575 generic.go:356] "Generic (PLEG): container finished" podID="28e88da1-b991-43f8-939f-b6104f678d9f" containerID="2677373b77bd4749a772466b15edeb7a2bfc5c6a15001d77a70c8eded5f363e2" exitCode=0 Jul 09 14:52:05.849548 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:05.849181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040319296" event={"ID":"28e88da1-b991-43f8-939f-b6104f678d9f","Type":"ContainerDied","Data":"2677373b77bd4749a772466b15edeb7a2bfc5c6a15001d77a70c8eded5f363e2"} Jul 09 14:52:05.886623 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:05.886590 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a62e69-5efb-4879-851a-35602807bc84" path="/var/lib/kubelet/pods/f3a62e69-5efb-4879-851a-35602807bc84/volumes" Jul 09 14:52:06.994526 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:06.994501 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:07.031478 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.031444 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7566-140598040319296_28e88da1-b991-43f8-939f-b6104f678d9f/test-curl-7566-140598040319296/0.log" Jul 09 14:52:07.066753 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.066716 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:07.069780 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.069754 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7566-140598040319296"] Jul 09 14:52:07.149601 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.149526 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7fds\" (UniqueName: \"kubernetes.io/projected/28e88da1-b991-43f8-939f-b6104f678d9f-kube-api-access-r7fds\") pod \"28e88da1-b991-43f8-939f-b6104f678d9f\" (UID: \"28e88da1-b991-43f8-939f-b6104f678d9f\") " Jul 09 14:52:07.151712 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.151687 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e88da1-b991-43f8-939f-b6104f678d9f-kube-api-access-r7fds" (OuterVolumeSpecName: "kube-api-access-r7fds") pod "28e88da1-b991-43f8-939f-b6104f678d9f" (UID: "28e88da1-b991-43f8-939f-b6104f678d9f"). InnerVolumeSpecName "kube-api-access-r7fds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:07.210809 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.210773 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:07.211315 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.211292 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e88da1-b991-43f8-939f-b6104f678d9f" containerName="test-curl-7566-140598040319296" Jul 09 14:52:07.211315 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.211315 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e88da1-b991-43f8-939f-b6104f678d9f" containerName="test-curl-7566-140598040319296" Jul 09 14:52:07.211465 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.211452 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e88da1-b991-43f8-939f-b6104f678d9f" containerName="test-curl-7566-140598040319296" Jul 09 14:52:07.214474 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.214457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:07.224291 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.224264 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:07.250162 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.250119 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7fds\" (UniqueName: \"kubernetes.io/projected/28e88da1-b991-43f8-939f-b6104f678d9f-kube-api-access-r7fds\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:07.350982 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.350947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bt7g\" (UniqueName: \"kubernetes.io/projected/063c14e9-e4c0-4225-8bc8-da70ff643431-kube-api-access-8bt7g\") pod \"test-curl-7566-140598040321168\" (UID: \"063c14e9-e4c0-4225-8bc8-da70ff643431\") " pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:07.452281 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.452172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bt7g\" (UniqueName: \"kubernetes.io/projected/063c14e9-e4c0-4225-8bc8-da70ff643431-kube-api-access-8bt7g\") pod \"test-curl-7566-140598040321168\" (UID: \"063c14e9-e4c0-4225-8bc8-da70ff643431\") " pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:07.467523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.467494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bt7g\" (UniqueName: \"kubernetes.io/projected/063c14e9-e4c0-4225-8bc8-da70ff643431-kube-api-access-8bt7g\") pod \"test-curl-7566-140598040321168\" (UID: \"063c14e9-e4c0-4225-8bc8-da70ff643431\") " pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:07.524121 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.524084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:07.668286 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.668260 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:07.670918 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:52:07.670874 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod063c14e9_e4c0_4225_8bc8_da70ff643431.slice/crio-76c70e1dda3a3acb47b3256342709651e8a79a028c5f95938ec7b936efef2fdd WatchSource:0}: Error finding container 76c70e1dda3a3acb47b3256342709651e8a79a028c5f95938ec7b936efef2fdd: Status 404 returned error can't find the container with id 76c70e1dda3a3acb47b3256342709651e8a79a028c5f95938ec7b936efef2fdd Jul 09 14:52:07.861493 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.861464 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20075af0febc08b6175a91586ba97a9167d5a11bc191b8ebf91cfc6e082a3063" Jul 09 14:52:07.861672 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.861516 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:07.862793 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.862757 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040321168" event={"ID":"063c14e9-e4c0-4225-8bc8-da70ff643431","Type":"ContainerStarted","Data":"76c70e1dda3a3acb47b3256342709651e8a79a028c5f95938ec7b936efef2fdd"} Jul 09 14:52:07.880167 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:52:07.880127 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7566-140598040319296\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="28e88da1-b991-43f8-939f-b6104f678d9f" pod="openshift-ingress/test-curl-7566-140598040319296" Jul 09 14:52:07.886583 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:07.886556 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e88da1-b991-43f8-939f-b6104f678d9f" path="/var/lib/kubelet/pods/28e88da1-b991-43f8-939f-b6104f678d9f/volumes" Jul 09 14:52:08.870381 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:08.870348 2575 generic.go:356] "Generic (PLEG): container finished" podID="063c14e9-e4c0-4225-8bc8-da70ff643431" containerID="6649903b4e453ed4bbf4fb0257c40366f21d48de1ac0b881bc9ece21205b099a" exitCode=0 Jul 09 14:52:08.870799 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:08.870426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040321168" event={"ID":"063c14e9-e4c0-4225-8bc8-da70ff643431","Type":"ContainerDied","Data":"6649903b4e453ed4bbf4fb0257c40366f21d48de1ac0b881bc9ece21205b099a"} Jul 09 14:52:10.011412 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.011379 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:10.052523 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.052495 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7566-140598040321168_063c14e9-e4c0-4225-8bc8-da70ff643431/test-curl-7566-140598040321168/0.log" Jul 09 14:52:10.089007 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.088968 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:10.102381 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.102353 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:10.182263 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.182152 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bt7g\" (UniqueName: \"kubernetes.io/projected/063c14e9-e4c0-4225-8bc8-da70ff643431-kube-api-access-8bt7g\") pod \"063c14e9-e4c0-4225-8bc8-da70ff643431\" (UID: \"063c14e9-e4c0-4225-8bc8-da70ff643431\") " Jul 09 14:52:10.184256 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.184217 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063c14e9-e4c0-4225-8bc8-da70ff643431-kube-api-access-8bt7g" (OuterVolumeSpecName: "kube-api-access-8bt7g") pod "063c14e9-e4c0-4225-8bc8-da70ff643431" (UID: "063c14e9-e4c0-4225-8bc8-da70ff643431"). InnerVolumeSpecName "kube-api-access-8bt7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:10.242896 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.242857 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:10.243316 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.243301 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063c14e9-e4c0-4225-8bc8-da70ff643431" containerName="test-curl-7566-140598040321168" Jul 09 14:52:10.243371 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.243317 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="063c14e9-e4c0-4225-8bc8-da70ff643431" containerName="test-curl-7566-140598040321168" Jul 09 14:52:10.243408 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.243390 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="063c14e9-e4c0-4225-8bc8-da70ff643431" containerName="test-curl-7566-140598040321168" Jul 09 14:52:10.246529 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.246512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:10.258772 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.258745 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:10.283134 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.283104 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bt7g\" (UniqueName: \"kubernetes.io/projected/063c14e9-e4c0-4225-8bc8-da70ff643431-kube-api-access-8bt7g\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:10.384027 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.383979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6xf\" (UniqueName: \"kubernetes.io/projected/d99462b1-3035-4730-b359-ecd3573158de-kube-api-access-ll6xf\") pod \"test-curl-7566-140598040321168\" (UID: \"d99462b1-3035-4730-b359-ecd3573158de\") " pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:10.484593 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.484494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6xf\" (UniqueName: \"kubernetes.io/projected/d99462b1-3035-4730-b359-ecd3573158de-kube-api-access-ll6xf\") pod \"test-curl-7566-140598040321168\" (UID: \"d99462b1-3035-4730-b359-ecd3573158de\") " pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:10.500151 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.500120 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6xf\" (UniqueName: \"kubernetes.io/projected/d99462b1-3035-4730-b359-ecd3573158de-kube-api-access-ll6xf\") pod \"test-curl-7566-140598040321168\" (UID: \"d99462b1-3035-4730-b359-ecd3573158de\") " pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:10.555277 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.555245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:10.705484 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.705409 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:10.707983 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:52:10.707955 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd99462b1_3035_4730_b359_ecd3573158de.slice/crio-a1a536be2ed770a11a2f7a12abeb911e2bd9541b3e24776940b9db79218a6949 WatchSource:0}: Error finding container a1a536be2ed770a11a2f7a12abeb911e2bd9541b3e24776940b9db79218a6949: Status 404 returned error can't find the container with id a1a536be2ed770a11a2f7a12abeb911e2bd9541b3e24776940b9db79218a6949 Jul 09 14:52:10.882513 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.882485 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:10.882666 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.882524 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c70e1dda3a3acb47b3256342709651e8a79a028c5f95938ec7b936efef2fdd" Jul 09 14:52:10.883826 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.883801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040321168" event={"ID":"d99462b1-3035-4730-b359-ecd3573158de","Type":"ContainerStarted","Data":"a1a536be2ed770a11a2f7a12abeb911e2bd9541b3e24776940b9db79218a6949"} Jul 09 14:52:10.894769 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:10.894734 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7566-140598040321168" oldPodUID="063c14e9-e4c0-4225-8bc8-da70ff643431" podUID="d99462b1-3035-4730-b359-ecd3573158de" Jul 09 14:52:11.887664 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:11.887627 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063c14e9-e4c0-4225-8bc8-da70ff643431" path="/var/lib/kubelet/pods/063c14e9-e4c0-4225-8bc8-da70ff643431/volumes" Jul 09 14:52:11.890755 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:11.890733 2575 generic.go:356] "Generic (PLEG): container finished" podID="d99462b1-3035-4730-b359-ecd3573158de" containerID="98b2a36cc0f058fa33fb0d67b808c517c583e377d8651232abec35f94d51c6fb" exitCode=0 Jul 09 14:52:11.890877 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:11.890802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7566-140598040321168" event={"ID":"d99462b1-3035-4730-b359-ecd3573158de","Type":"ContainerDied","Data":"98b2a36cc0f058fa33fb0d67b808c517c583e377d8651232abec35f94d51c6fb"} Jul 09 14:52:13.048457 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.048427 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:13.075845 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.075818 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7566-140598040321168_d99462b1-3035-4730-b359-ecd3573158de/test-curl-7566-140598040321168/0.log" Jul 09 14:52:13.102918 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.102879 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:13.104874 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.104854 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7566-140598040321168"] Jul 09 14:52:13.211778 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.211693 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll6xf\" (UniqueName: \"kubernetes.io/projected/d99462b1-3035-4730-b359-ecd3573158de-kube-api-access-ll6xf\") pod \"d99462b1-3035-4730-b359-ecd3573158de\" (UID: \"d99462b1-3035-4730-b359-ecd3573158de\") " Jul 09 14:52:13.213767 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.213741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99462b1-3035-4730-b359-ecd3573158de-kube-api-access-ll6xf" (OuterVolumeSpecName: "kube-api-access-ll6xf") pod "d99462b1-3035-4730-b359-ecd3573158de" (UID: "d99462b1-3035-4730-b359-ecd3573158de"). InnerVolumeSpecName "kube-api-access-ll6xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:13.315346 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.315313 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6xf\" (UniqueName: \"kubernetes.io/projected/d99462b1-3035-4730-b359-ecd3573158de-kube-api-access-ll6xf\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:13.887901 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.887866 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d99462b1-3035-4730-b359-ecd3573158de" path="/var/lib/kubelet/pods/d99462b1-3035-4730-b359-ecd3573158de/volumes" Jul 09 14:52:13.902350 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.902324 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7566-140598040321168" Jul 09 14:52:13.902494 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:13.902352 2575 scope.go:117] "RemoveContainer" containerID="98b2a36cc0f058fa33fb0d67b808c517c583e377d8651232abec35f94d51c6fb" Jul 09 14:52:21.010623 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:21.010586 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s"] Jul 09 14:52:21.011159 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:21.010803 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" containerID="cri-o://8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7" gracePeriod=30 Jul 09 14:52:22.226415 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.226377 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt"] Jul 09 14:52:22.226836 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.226644 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" podUID="88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" containerName="maas-api" containerID="cri-o://82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a" gracePeriod=30 Jul 09 14:52:22.506004 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.505977 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:52:22.588168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.588138 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrhsq\" (UniqueName: \"kubernetes.io/projected/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-kube-api-access-qrhsq\") pod \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " Jul 09 14:52:22.588377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.588185 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-maas-api-tls\") pod \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " Jul 09 14:52:22.588377 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.588203 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-openshift-service-ca\") pod \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\" (UID: \"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55\") " Jul 09 14:52:22.588637 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.588612 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" (UID: "88f0c3e6-66b7-41e7-8f52-97a94f7b1b55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:52:22.590432 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.590401 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" (UID: "88f0c3e6-66b7-41e7-8f52-97a94f7b1b55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:52:22.590432 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.590411 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-kube-api-access-qrhsq" (OuterVolumeSpecName: "kube-api-access-qrhsq") pod "88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" (UID: "88f0c3e6-66b7-41e7-8f52-97a94f7b1b55"). InnerVolumeSpecName "kube-api-access-qrhsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:22.689408 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.689375 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrhsq\" (UniqueName: \"kubernetes.io/projected/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-kube-api-access-qrhsq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:22.689408 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.689403 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:22.689408 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.689416 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:22.954957 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.954921 2575 generic.go:356] "Generic (PLEG): container finished" podID="88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" containerID="82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a" exitCode=0 Jul 09 14:52:22.955156 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.954964 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" Jul 09 14:52:22.955156 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.954998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" event={"ID":"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55","Type":"ContainerDied","Data":"82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a"} Jul 09 14:52:22.955156 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.955050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt" event={"ID":"88f0c3e6-66b7-41e7-8f52-97a94f7b1b55","Type":"ContainerDied","Data":"42340e62d71026cdb3d750fc591842a9194718e462b4afd6af1026d8fe7ccd7f"} Jul 09 14:52:22.955156 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.955072 2575 scope.go:117] "RemoveContainer" containerID="82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a" Jul 09 14:52:22.964675 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.964652 2575 scope.go:117] "RemoveContainer" containerID="82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a" Jul 09 14:52:22.964990 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:52:22.964972 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a\": container with ID starting with 82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a not found: ID does not exist" containerID="82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a" Jul 09 14:52:22.965040 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.964999 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a"} err="failed to get container status \"82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a\": rpc error: code = NotFound desc = could not find container \"82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a\": container with ID starting with 82522468e9659159ed7583ba322ecef8a769e87c7e1ca361f28455c0abe3826a not found: ID does not exist" Jul 09 14:52:22.983042 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.983006 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt"] Jul 09 14:52:22.985562 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:22.985535 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c4eda7-568574597f-qrgtt"] Jul 09 14:52:23.888267 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:23.888209 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" path="/var/lib/kubelet/pods/88f0c3e6-66b7-41e7-8f52-97a94f7b1b55/volumes" Jul 09 14:52:25.698706 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:25.698667 2575 patch_prober.go:28] interesting pod/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 14:52:25.699089 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:25.698737 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 14:52:26.281836 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.281763 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:52:26.315265 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315217 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-credential-socket\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315443 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315344 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-certs\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315443 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315368 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istiod-ca-cert\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315443 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315392 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-token\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315443 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315415 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-podinfo\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315469 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnztp\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-kube-api-access-mnztp\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315470 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:26.315671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-socket\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315517 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-data\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315671 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-envoy\") pod \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\" (UID: \"fdc8eb6a-b410-4522-8c07-03dde0e4e028\") " Jul 09 14:52:26.315942 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:26.315942 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315898 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.315942 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.315914 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-credential-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.316122 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.316091 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:26.316198 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.316119 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:52:26.316711 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.316677 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-data" (OuterVolumeSpecName: "istio-data") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:26.318276 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.318253 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:52:26.318276 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.318257 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:26.318586 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.318566 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-token" (OuterVolumeSpecName: "istio-token") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:26.318653 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.318584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-kube-api-access-mnztp" (OuterVolumeSpecName: "kube-api-access-mnztp") pod "fdc8eb6a-b410-4522-8c07-03dde0e4e028" (UID: "fdc8eb6a-b410-4522-8c07-03dde0e4e028"). InnerVolumeSpecName "kube-api-access-mnztp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:26.416892 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.416857 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istiod-ca-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.416892 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.416885 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-token\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.416892 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.416894 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-podinfo\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.416892 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.416902 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnztp\" (UniqueName: \"kubernetes.io/projected/fdc8eb6a-b410-4522-8c07-03dde0e4e028-kube-api-access-mnztp\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.417168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.416912 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-workload-socket\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.417168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.416920 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-data\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.417168 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.416928 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc8eb6a-b410-4522-8c07-03dde0e4e028-istio-envoy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\"" Jul 09 14:52:26.979644 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.979602 2575 generic.go:356] "Generic (PLEG): container finished" podID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerID="8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7" exitCode=0 Jul 09 14:52:26.980127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.979649 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" Jul 09 14:52:26.980127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.979644 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" event={"ID":"fdc8eb6a-b410-4522-8c07-03dde0e4e028","Type":"ContainerDied","Data":"8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7"} Jul 09 14:52:26.980127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.979783 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s" event={"ID":"fdc8eb6a-b410-4522-8c07-03dde0e4e028","Type":"ContainerDied","Data":"1dad508191dd4207b7be229d96787f5f980c3d09017c04b8886699e8246c0a00"} Jul 09 14:52:26.980127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.979809 2575 scope.go:117] "RemoveContainer" containerID="8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7" Jul 09 14:52:26.989701 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.989683 2575 scope.go:117] "RemoveContainer" containerID="d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd" Jul 09 14:52:26.997908 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.997889 2575 scope.go:117] "RemoveContainer" containerID="8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7" Jul 09 14:52:26.998170 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:52:26.998151 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7\": container with ID starting with 8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7 not found: ID does not exist" containerID="8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7" Jul 09 14:52:26.998258 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.998182 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7"} err="failed to get container status \"8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7\": rpc error: code = NotFound desc = could not find container \"8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7\": container with ID starting with 8c7c8ba826e82c9cb0c2490d5cb9285c81d561981911367dc89dbb9d632322b7 not found: ID does not exist" Jul 09 14:52:26.998258 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.998206 2575 scope.go:117] "RemoveContainer" containerID="d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd" Jul 09 14:52:26.998488 ip-10-0-141-45 kubenswrapper[2575]: E0709 14:52:26.998467 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd\": container with ID starting with d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd not found: ID does not exist" containerID="d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd" Jul 09 14:52:26.998528 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:26.998496 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd"} err="failed to get container status \"d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd\": rpc error: code = NotFound desc = could not find container \"d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd\": container with ID starting with d939f39aeff90a87c100ca727abc21e32a59e59e08c3661a0c65788e7ffa6bbd not found: ID does not exist" Jul 09 14:52:27.003165 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:27.003143 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s"] Jul 09 14:52:27.012099 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:27.012075 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b22ad3-openshift-default-5666d7f4cf-47p2s"] Jul 09 14:52:27.886425 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:27.886393 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" path="/var/lib/kubelet/pods/fdc8eb6a-b410-4522-8c07-03dde0e4e028/volumes" Jul 09 14:52:34.224964 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:34.224931 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kvzvv_73951f2a-1722-4654-85f3-908da3a1a9ba/manager/0.log" Jul 09 14:52:34.570648 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:34.570619 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k2bnn_f69b456e-1e0c-4b05-9a68-1953513ba62f/server/0.log" Jul 09 14:52:34.685870 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:34.685836 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fk49v_aa36c248-3d76-4fe6-b2ce-a237aac91a2b/manager/0.log" Jul 09 14:52:34.806331 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:34.806302 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-6f4fv_19b30445-5808-421c-a24c-715d93a47cb8/copy-manifests/0.log" Jul 09 14:52:34.813592 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:34.813569 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-6f4fv_19b30445-5808-421c-a24c-715d93a47cb8/manager/0.log" Jul 09 14:52:35.530972 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:35.530941 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9tc6p_eba1615b-ea0b-4afc-8b15-98b8eb48ada5/postgres/0.log" Jul 09 14:52:36.946240 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:36.946191 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xxpw5_3edf0e79-eea6-4861-9d8e-643621d944dc/manager/0.log" Jul 09 14:52:37.307217 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:37.307189 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zpvfx_7ac96cb2-4d04-4eeb-a2f9-276d62db5e31/registry-server/0.log" Jul 09 14:52:37.423977 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:37.423941 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cmn45_c8509b3b-9f7c-4dfc-a688-822903dd79ba/manager/0.log" Jul 09 14:52:37.662095 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:37.662002 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cfdpx_ce0708cc-7027-4f90-939c-0478520ae475/manager/0.log" Jul 09 14:52:38.385634 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:38.385600 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-8zwxm_d09296b9-b36a-4945-88be-6f70490e6e8c/kube-auth-proxy/0.log" Jul 09 14:52:38.864626 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:38.864593 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74555698d-dkf2p_7e281907-2c2d-4944-8736-22f20ce41bb7/router/0.log" Jul 09 14:52:39.221200 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:39.221124 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5nlhl_fc3a77b9-7878-49bf-bcb2-e2f429361bef/main/0.log" Jul 09 14:52:39.449988 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:39.449956 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nhtwk_e863a573-9ead-45ad-a14b-7af95a8f10f7/main/0.log" Jul 09 14:52:43.908472 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908394 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mpppr/must-gather-vf5qj"] Jul 09 14:52:43.908853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908759 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" Jul 09 14:52:43.908853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908769 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" Jul 09 14:52:43.908853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908777 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" containerName="maas-api" Jul 09 14:52:43.908853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908783 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" containerName="maas-api" Jul 09 14:52:43.908853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908790 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99462b1-3035-4730-b359-ecd3573158de" containerName="test-curl-7566-140598040321168" Jul 09 14:52:43.908853 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908796 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99462b1-3035-4730-b359-ecd3573158de" containerName="test-curl-7566-140598040321168" Jul 09 14:52:43.909039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908872 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d99462b1-3035-4730-b359-ecd3573158de" containerName="test-curl-7566-140598040321168" Jul 09 14:52:43.909039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908882 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" Jul 09 14:52:43.909039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908889 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f0c3e6-66b7-41e7-8f52-97a94f7b1b55" containerName="maas-api" Jul 09 14:52:43.909039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908896 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" Jul 09 14:52:43.909039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908963 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" Jul 09 14:52:43.909039 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.908969 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc8eb6a-b410-4522-8c07-03dde0e4e028" containerName="istio-proxy" Jul 09 14:52:43.912103 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.912086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:43.916037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.916004 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mpppr\"/\"openshift-service-ca.crt\"" Jul 09 14:52:43.916037 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.916035 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mpppr\"/\"default-dockercfg-mbnb2\"" Jul 09 14:52:43.916243 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.916158 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mpppr\"/\"kube-root-ca.crt\"" Jul 09 14:52:43.927273 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.927245 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mpppr/must-gather-vf5qj"] Jul 09 14:52:43.944215 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.944182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/faffade5-9cfc-45c7-ad7c-35a231d1cec5-must-gather-output\") pod \"must-gather-vf5qj\" (UID: \"faffade5-9cfc-45c7-ad7c-35a231d1cec5\") " pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:43.944435 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:43.944258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmdqf\" (UniqueName: \"kubernetes.io/projected/faffade5-9cfc-45c7-ad7c-35a231d1cec5-kube-api-access-xmdqf\") pod \"must-gather-vf5qj\" (UID: \"faffade5-9cfc-45c7-ad7c-35a231d1cec5\") " pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:44.045103 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:44.045068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/faffade5-9cfc-45c7-ad7c-35a231d1cec5-must-gather-output\") pod \"must-gather-vf5qj\" (UID: \"faffade5-9cfc-45c7-ad7c-35a231d1cec5\") " pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:44.045103 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:44.045110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmdqf\" (UniqueName: \"kubernetes.io/projected/faffade5-9cfc-45c7-ad7c-35a231d1cec5-kube-api-access-xmdqf\") pod \"must-gather-vf5qj\" (UID: \"faffade5-9cfc-45c7-ad7c-35a231d1cec5\") " pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:44.045453 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:44.045430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/faffade5-9cfc-45c7-ad7c-35a231d1cec5-must-gather-output\") pod \"must-gather-vf5qj\" (UID: \"faffade5-9cfc-45c7-ad7c-35a231d1cec5\") " pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:44.054362 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:44.054336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmdqf\" (UniqueName: \"kubernetes.io/projected/faffade5-9cfc-45c7-ad7c-35a231d1cec5-kube-api-access-xmdqf\") pod \"must-gather-vf5qj\" (UID: \"faffade5-9cfc-45c7-ad7c-35a231d1cec5\") " pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:44.221844 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:44.221739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mpppr/must-gather-vf5qj" Jul 09 14:52:44.359950 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:44.359766 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mpppr/must-gather-vf5qj"] Jul 09 14:52:44.362761 ip-10-0-141-45 kubenswrapper[2575]: W0709 14:52:44.362729 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaffade5_9cfc_45c7_ad7c_35a231d1cec5.slice/crio-3fbc106c65437e65dd5c04419ed5bfbfbf9d3d7242b08061f1468d6dec0299cd WatchSource:0}: Error finding container 3fbc106c65437e65dd5c04419ed5bfbfbf9d3d7242b08061f1468d6dec0299cd: Status 404 returned error can't find the container with id 3fbc106c65437e65dd5c04419ed5bfbfbf9d3d7242b08061f1468d6dec0299cd Jul 09 14:52:45.080636 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:45.080598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mpppr/must-gather-vf5qj" event={"ID":"faffade5-9cfc-45c7-ad7c-35a231d1cec5","Type":"ContainerStarted","Data":"3fbc106c65437e65dd5c04419ed5bfbfbf9d3d7242b08061f1468d6dec0299cd"} Jul 09 14:52:46.088550 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:46.088452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mpppr/must-gather-vf5qj" event={"ID":"faffade5-9cfc-45c7-ad7c-35a231d1cec5","Type":"ContainerStarted","Data":"121695dc1d37f7b151469ce6916de02169bbe24dc87bc632f86511e68fc56ec1"} Jul 09 14:52:46.088550 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:46.088497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mpppr/must-gather-vf5qj" event={"ID":"faffade5-9cfc-45c7-ad7c-35a231d1cec5","Type":"ContainerStarted","Data":"ecc325b78fb6a0e3d017e6424f82961be5936af93521e2dd519780803edc4194"} Jul 09 14:52:46.107587 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:46.107521 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mpppr/must-gather-vf5qj" podStartSLOduration=2.21842194 podStartE2EDuration="3.107499229s" totalImagesPullingTime="889.077289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:46.105066121 +0000 UTC m=+2332.793995371" watchObservedRunningTime="2026-07-09 14:52:46.107499229 +0000 UTC m=+2332.796428470" Jul 09 14:52:46.961990 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:46.961961 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bgmqg_2cdcb585-98f7-4c44-928e-68b3dc41164e/global-pull-secret-syncer/0.log" Jul 09 14:52:47.117867 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:47.117841 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d22n5_86ae436a-86e2-4812-86cd-7ca9b4adc1b5/konnectivity-agent/0.log" Jul 09 14:52:47.175668 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:47.175638 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-45.ec2.internal_a1c7bc3b4c133f1b0578384be47f8d38/haproxy/0.log" Jul 09 14:52:52.427186 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:52.427157 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xxpw5_3edf0e79-eea6-4861-9d8e-643621d944dc/manager/0.log" Jul 09 14:52:52.549127 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:52.549089 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zpvfx_7ac96cb2-4d04-4eeb-a2f9-276d62db5e31/registry-server/0.log" Jul 09 14:52:52.682640 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:52.682464 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cmn45_c8509b3b-9f7c-4dfc-a688-822903dd79ba/manager/0.log" Jul 09 14:52:52.791876 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:52.791830 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cfdpx_ce0708cc-7027-4f90-939c-0478520ae475/manager/0.log" Jul 09 14:52:54.749984 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:54.749947 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-24tn7_a9284783-b78b-4612-96b7-38fb17d6e4b4/cluster-monitoring-operator/0.log" Jul 09 14:52:54.934471 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:54.934388 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5k8cm_818beee3-d1d7-495b-82db-bd1c0559a006/node-exporter/0.log" Jul 09 14:52:54.960970 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:54.960927 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5k8cm_818beee3-d1d7-495b-82db-bd1c0559a006/kube-rbac-proxy/0.log" Jul 09 14:52:54.984962 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:54.984923 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5k8cm_818beee3-d1d7-495b-82db-bd1c0559a006/init-textfile/0.log" Jul 09 14:52:55.472204 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:55.472173 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ls2lj_101345db-9986-43e2-acf5-856f6e0f6610/prometheus-operator/0.log" Jul 09 14:52:55.496096 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:55.496068 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ls2lj_101345db-9986-43e2-acf5-856f6e0f6610/kube-rbac-proxy/0.log" Jul 09 14:52:55.531166 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:55.531139 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-jbhm2_4ae5f574-4d81-4450-a29d-4fb74f3626b9/prometheus-operator-admission-webhook/0.log" Jul 09 14:52:56.696386 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.696345 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4"] Jul 09 14:52:56.704823 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.704789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.709712 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.709674 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4"] Jul 09 14:52:56.802008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.801690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-sys\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.802008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.801757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-lib-modules\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.802008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.801828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-podres\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.802008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.801868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-proc\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.802008 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.801925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d85nw\" (UniqueName: \"kubernetes.io/projected/c2c79429-99f1-48c7-8be5-8c90c97257fd-kube-api-access-d85nw\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903002 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.902970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-podres\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903002 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-proc\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-proc\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d85nw\" (UniqueName: \"kubernetes.io/projected/c2c79429-99f1-48c7-8be5-8c90c97257fd-kube-api-access-d85nw\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-podres\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-sys\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903289 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-lib-modules\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903565 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-sys\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.903565 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.903406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2c79429-99f1-48c7-8be5-8c90c97257fd-lib-modules\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:56.913782 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:56.913752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d85nw\" (UniqueName: \"kubernetes.io/projected/c2c79429-99f1-48c7-8be5-8c90c97257fd-kube-api-access-d85nw\") pod \"perf-node-gather-daemonset-mnww4\" (UID: \"c2c79429-99f1-48c7-8be5-8c90c97257fd\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:57.018914 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:57.018827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:57.048594 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:57.047786 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-866jh_967aa6b7-9ffe-405d-93c6-fdbc4c9a3715/networking-console-plugin/0.log" Jul 09 14:52:57.228373 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:57.228317 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4"] Jul 09 14:52:57.691441 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:57.691366 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/2.log" Jul 09 14:52:57.697575 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:57.697540 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wxw4j_66bca9b3-d1e5-475b-80a2-824250d8ae0f/console-operator/3.log" Jul 09 14:52:58.180412 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:58.180371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" event={"ID":"c2c79429-99f1-48c7-8be5-8c90c97257fd","Type":"ContainerStarted","Data":"79688fc3e77099582f08e8cfedb484fe0b96bb33d8632218aa9e58f30a90dc50"} Jul 09 14:52:58.180695 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:58.180435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" event={"ID":"c2c79429-99f1-48c7-8be5-8c90c97257fd","Type":"ContainerStarted","Data":"318a298d7e11ba14d7b0399295da58dc9396d5cb2dea29e8a03b6e1ab31e3cc6"} Jul 09 14:52:58.180695 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:58.180608 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:52:58.197870 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:58.197822 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" podStartSLOduration=2.197807912 podStartE2EDuration="2.197807912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:58.196453086 +0000 UTC m=+2344.885382347" watchObservedRunningTime="2026-07-09 14:52:58.197807912 +0000 UTC m=+2344.886737151" Jul 09 14:52:58.212532 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:58.212503 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-mtpth_ec42a2b9-ace3-48cd-b822-411e206e717d/download-server/0.log" Jul 09 14:52:58.801648 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:58.801592 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-pkfdk_7f992406-0c0a-420d-872d-b3a3dd9448a5/volume-data-source-validator/0.log" Jul 09 14:52:59.879400 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:59.879371 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8fdl_94d25ba2-c533-44bd-be10-c9da17f3929c/dns/0.log" Jul 09 14:52:59.907159 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:52:59.907123 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8fdl_94d25ba2-c533-44bd-be10-c9da17f3929c/kube-rbac-proxy/0.log" Jul 09 14:53:00.040706 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:00.040681 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fdr6_aad276ab-66e3-4cc8-a0a5-6749b625419b/dns-node-resolver/0.log" Jul 09 14:53:00.681433 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:00.681403 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9cbwm_0115aff2-b98a-4415-8204-4de41320eb9a/node-ca/0.log" Jul 09 14:53:01.881460 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:01.881432 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-8zwxm_d09296b9-b36a-4945-88be-6f70490e6e8c/kube-auth-proxy/0.log" Jul 09 14:53:02.048765 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:02.048733 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74555698d-dkf2p_7e281907-2c2d-4944-8736-22f20ce41bb7/router/0.log" Jul 09 14:53:02.807711 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:02.807671 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sglr8_1055a6aa-997c-447b-ab11-452a2639dbd8/serve-healthcheck-canary/0.log" Jul 09 14:53:03.340288 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:03.340218 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qptsb_5a3562bb-92a7-4cc1-91bf-9df604d84f92/insights-operator/0.log" Jul 09 14:53:03.341895 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:03.341868 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qptsb_5a3562bb-92a7-4cc1-91bf-9df604d84f92/insights-operator/1.log" Jul 09 14:53:03.517561 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:03.517534 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2gbj_14e7daa9-1b76-445a-9b5a-b3106f89ff9d/kube-rbac-proxy/0.log" Jul 09 14:53:03.542440 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:03.542410 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2gbj_14e7daa9-1b76-445a-9b5a-b3106f89ff9d/exporter/0.log" Jul 09 14:53:03.568138 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:03.568107 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2gbj_14e7daa9-1b76-445a-9b5a-b3106f89ff9d/extractor/0.log" Jul 09 14:53:04.199033 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:04.199002 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-mnww4" Jul 09 14:53:05.978176 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:05.978149 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kvzvv_73951f2a-1722-4654-85f3-908da3a1a9ba/manager/0.log" Jul 09 14:53:06.226042 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:06.225962 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k2bnn_f69b456e-1e0c-4b05-9a68-1953513ba62f/server/0.log" Jul 09 14:53:06.310054 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:06.310014 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fk49v_aa36c248-3d76-4fe6-b2ce-a237aac91a2b/manager/0.log" Jul 09 14:53:06.352212 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:06.352157 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-6f4fv_19b30445-5808-421c-a24c-715d93a47cb8/manager/0.log" Jul 09 14:53:06.382102 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:06.382075 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-6f4fv_19b30445-5808-421c-a24c-715d93a47cb8/copy-manifests/0.log" Jul 09 14:53:07.577031 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:07.577005 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-8zwxm_d09296b9-b36a-4945-88be-6f70490e6e8c/kube-auth-proxy/0.log" Jul 09 14:53:07.713363 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:07.713332 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74555698d-dkf2p_7e281907-2c2d-4944-8736-22f20ce41bb7/router/0.log" Jul 09 14:53:09.210496 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:09.210467 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-8zwxm_d09296b9-b36a-4945-88be-6f70490e6e8c/kube-auth-proxy/0.log" Jul 09 14:53:09.464766 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:09.464698 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74555698d-dkf2p_7e281907-2c2d-4944-8736-22f20ce41bb7/router/0.log" Jul 09 14:53:13.040971 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:13.040934 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-zzdzv_7a0f826f-d4ab-4b28-bda1-ba9f9a674b65/migrator/0.log" Jul 09 14:53:13.067242 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:13.067188 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-zzdzv_7a0f826f-d4ab-4b28-bda1-ba9f9a674b65/graceful-termination/0.log" Jul 09 14:53:13.433919 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:13.432724 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-8jz5b_7d6410bb-ed3b-480e-bd9f-1bdef3e5e743/kube-storage-version-migrator-operator/1.log" Jul 09 14:53:13.433919 ip-10-0-141-45 kubenswrapper[2575]: I0709 14:53:13.433737 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-8jz5b_7d6410bb-ed3b-480e-bd9f-1bdef3e5e743/kube-storage-version-migrator-operator/0.log"