Jul 10 18:01:00.093457 ip-10-0-141-217 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 18:01:00.093472 ip-10-0-141-217 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 18:01:00.093482 ip-10-0-141-217 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 18:01:00.093792 ip-10-0-141-217 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 18:01:10.135658 ip-10-0-141-217 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 18:01:10.135680 ip-10-0-141-217 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 47312c9c0bb143e7ab96108d090c4962 -- Jul 10 18:03:10.779890 ip-10-0-141-217 systemd[1]: Starting Kubernetes Kubelet... Jul 10 18:03:11.202703 ip-10-0-141-217 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:03:11.202703 ip-10-0-141-217 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 18:03:11.202703 ip-10-0-141-217 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:03:11.202703 ip-10-0-141-217 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 18:03:11.202703 ip-10-0-141-217 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:03:11.205491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.205389 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 18:03:11.210549 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210527 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:03:11.210549 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210545 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:03:11.210549 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210551 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210557 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210561 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210565 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210569 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210573 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210577 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210581 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210585 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210589 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210593 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210614 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210619 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210623 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210627 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210632 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210636 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210640 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210646 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:03:11.210773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210660 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210664 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210668 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210673 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210678 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210683 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210688 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210692 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210698 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210703 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210708 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210712 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210717 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210721 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210725 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210730 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210735 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210740 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210745 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:03:11.211262 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210750 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210754 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210758 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210763 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210767 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210771 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210775 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210780 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210784 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210788 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210792 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210796 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210801 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210805 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:03:11.211738 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.210809 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:03:11.212144 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212131 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212148 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212154 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212162 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212168 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212172 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212177 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212182 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212186 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212191 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:03:11.212198 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212196 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212204 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212208 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212213 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212217 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212221 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212228 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212232 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212236 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212239 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212244 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212249 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212253 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212257 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212261 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212266 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212271 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212276 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212281 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212285 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:03:11.212694 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212289 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212294 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212298 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212302 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212307 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212311 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212315 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212320 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212324 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212329 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212333 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212339 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212345 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:03:11.213488 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.212350 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213541 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213551 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213556 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213561 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213565 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213574 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213580 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213585 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213589 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213614 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213621 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213626 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213630 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213635 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213639 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213643 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213647 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213652 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:03:11.214063 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213657 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213660 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213665 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213669 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213673 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213677 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213681 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213685 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213689 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213693 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213698 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213701 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213706 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213710 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213714 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213722 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213729 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213733 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213737 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:03:11.214871 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213742 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213747 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213753 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213757 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213762 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213766 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213770 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213774 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213778 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213785 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213791 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213796 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213801 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213805 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213810 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213815 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213819 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213824 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213828 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213833 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:03:11.215716 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213837 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213841 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213845 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213849 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213853 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213857 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213861 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213865 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213871 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213876 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213882 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213886 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213890 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213894 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213899 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213903 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213907 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213911 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213916 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213920 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:03:11.216572 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213924 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213928 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213932 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213937 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213941 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213945 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213949 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213953 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213957 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213962 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213966 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213970 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213974 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213978 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213982 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213986 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213990 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213994 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.213998 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:03:11.217181 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214002 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214008 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214012 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214743 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214753 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214758 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214763 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214768 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214773 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214778 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214782 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214786 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214790 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214795 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214799 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214804 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214808 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214812 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214816 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214821 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214825 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:03:11.217773 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214829 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214833 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214838 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214842 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214847 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214852 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214856 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214860 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214864 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214868 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214872 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214876 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214881 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214885 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214889 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214893 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214897 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214901 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214905 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214910 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:03:11.218338 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214914 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214919 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214923 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214928 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214932 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214936 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214940 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214944 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214948 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214952 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214959 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214965 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214970 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214976 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214981 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214987 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214992 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.214998 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:03:11.219066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215005 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215011 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215016 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215020 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215025 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215030 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215035 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215042 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215046 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215051 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215055 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215059 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215064 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215068 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215073 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215077 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215081 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215085 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215090 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215094 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:03:11.219563 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215098 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215102 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215106 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215110 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215114 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215118 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215122 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215127 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215131 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215136 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215140 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215145 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215149 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215153 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215157 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215162 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215166 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215171 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215182 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:03:11.220073 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215187 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215191 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215195 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.215199 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215302 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215313 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215324 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215331 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215338 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215344 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215350 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215357 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215362 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215367 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215373 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215378 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215383 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215388 2570 flags.go:64] FLAG: --cgroup-root="" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215392 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215397 2570 flags.go:64] FLAG: --client-ca-file="" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215401 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215406 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215413 2570 flags.go:64] FLAG: --cluster-domain="" Jul 10 18:03:11.220524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215417 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215423 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215428 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215433 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215440 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215444 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215450 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215455 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215460 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215465 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215470 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215475 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215480 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215486 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215492 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215497 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215502 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215507 2570 flags.go:64] FLAG: --enable-server="true" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215512 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215519 2570 flags.go:64] FLAG: --event-burst="100" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215524 2570 flags.go:64] FLAG: --event-qps="50" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215529 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215534 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215539 2570 flags.go:64] FLAG: --eviction-hard="" Jul 10 18:03:11.221118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215545 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215550 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215555 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215560 2570 flags.go:64] FLAG: --eviction-soft="" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215564 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215569 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215574 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215579 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215584 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215589 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215616 2570 flags.go:64] FLAG: --feature-gates="" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215623 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215628 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215633 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215638 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215642 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215647 2570 flags.go:64] FLAG: --help="false" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215652 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215657 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215662 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215667 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215673 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215679 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215684 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 18:03:11.221823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215689 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215693 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215698 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215704 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215709 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215714 2570 flags.go:64] FLAG: --kube-reserved="" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215719 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215723 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215727 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215732 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215736 2570 flags.go:64] FLAG: --lock-file="" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215741 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215745 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215751 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215762 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215769 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215774 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215779 2570 flags.go:64] FLAG: --logging-format="text" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215783 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215788 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215793 2570 flags.go:64] FLAG: --manifest-url="" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215798 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215805 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215810 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215816 2570 flags.go:64] FLAG: --max-pods="110" Jul 10 18:03:11.222423 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215821 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215826 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215830 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215835 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215839 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215844 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215849 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215861 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215866 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215870 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215875 2570 flags.go:64] FLAG: --pod-cidr="" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215881 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215890 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215895 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215900 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215904 2570 flags.go:64] FLAG: --port="10250" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215909 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215914 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d84bf0052898777a" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215919 2570 flags.go:64] FLAG: --qos-reserved="" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215924 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215929 2570 flags.go:64] FLAG: --register-node="true" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215933 2570 flags.go:64] FLAG: --register-with-taints="" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215939 2570 flags.go:64] FLAG: --registry-burst="10" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215945 2570 flags.go:64] FLAG: --registry-qps="5" Jul 10 18:03:11.223110 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215952 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215956 2570 flags.go:64] FLAG: --reserved-memory="" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215962 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215968 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215972 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215977 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215982 2570 flags.go:64] FLAG: --runonce="false" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215986 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215991 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.215996 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216001 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216005 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216010 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216015 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216020 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216025 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216029 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216034 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216038 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216043 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216049 2570 flags.go:64] FLAG: --system-cgroups="" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216054 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216063 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216068 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216072 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 18:03:11.223717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216079 2570 flags.go:64] FLAG: --tls-min-version="" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216083 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216088 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216093 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216097 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216103 2570 flags.go:64] FLAG: --v="2" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216110 2570 flags.go:64] FLAG: --version="false" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216122 2570 flags.go:64] FLAG: --vmodule="" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216128 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.216134 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216326 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216333 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216339 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216343 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216348 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216352 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216356 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216360 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216365 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216370 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216374 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216378 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:03:11.224319 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216382 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216387 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216391 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216395 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216400 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216404 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216409 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216413 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216418 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216422 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216426 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216431 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216437 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216444 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216451 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216456 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216462 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216467 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216471 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:03:11.224887 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216476 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216480 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216484 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216490 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216496 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216500 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216505 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216509 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216513 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216517 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216521 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216526 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216530 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216535 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216539 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216543 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216547 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216551 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216556 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216560 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:03:11.225702 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216565 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216569 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216573 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216577 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216580 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216584 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216588 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216611 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216616 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216622 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216628 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216632 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216636 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216641 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216645 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216649 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216653 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216657 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216662 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216666 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:03:11.226565 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216670 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216674 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216678 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216683 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216688 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216692 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216696 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216700 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216705 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216709 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216713 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216717 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216721 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216730 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216735 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216739 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216743 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216747 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216751 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:03:11.227493 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216755 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216759 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216765 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216771 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216775 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216779 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216783 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216787 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.216791 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.217615 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.227672 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.227693 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.227719 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.227728 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227808 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:03:11.228452 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227816 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227821 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227826 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227831 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227835 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227840 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227844 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227848 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227852 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227857 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227861 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227865 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227869 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227873 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227880 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227886 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227891 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227895 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227899 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:03:11.229113 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227903 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227909 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227913 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227917 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227922 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227926 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227930 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227935 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227940 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227945 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227949 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227953 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227957 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227962 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227966 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227970 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227974 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227978 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227982 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227986 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:03:11.229663 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227990 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227995 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.227999 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228003 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228007 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228011 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228016 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228020 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228024 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228028 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228032 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228036 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228040 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228045 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228049 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228054 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228059 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228063 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228067 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228071 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:03:11.230353 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228075 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228080 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228084 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228089 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228093 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228097 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228101 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228105 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228109 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228116 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228122 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228127 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228134 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228140 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228145 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228149 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228154 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228159 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:03:11.231236 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228163 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228167 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228171 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228175 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228179 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228183 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228188 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228192 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228197 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228201 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228206 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228210 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228214 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228218 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228222 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228227 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228231 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228236 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228240 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228244 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:03:11.232071 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228248 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.228257 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228434 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228444 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228448 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228452 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228457 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228461 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228465 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228469 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228473 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228477 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228482 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228486 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228490 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:03:11.232732 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228495 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228501 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228505 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228509 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228514 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228519 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228523 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228527 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228532 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228536 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228540 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228545 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228549 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228553 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228557 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228564 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228570 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228575 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228579 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228583 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:03:11.233111 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228587 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228591 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228617 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228621 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228625 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228629 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228634 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228640 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228646 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228651 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228656 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228661 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228665 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228669 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228673 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228677 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228681 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228685 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228690 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228694 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:03:11.233655 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228698 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228702 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228706 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228710 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228713 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228718 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228722 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228726 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228731 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228735 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228739 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228743 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228747 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228751 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228755 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228759 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228763 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228767 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228771 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228775 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:03:11.234148 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228779 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228788 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228794 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228798 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228803 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228807 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228811 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228816 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228820 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228824 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228829 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228833 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228837 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228841 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228845 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228849 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228853 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228858 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228862 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:03:11.234741 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228866 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228871 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228876 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228880 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228884 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228888 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:11.228892 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.228900 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.229087 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.232841 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.234261 2570 server.go:1019] "Starting client certificate rotation" Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.234368 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:03:11.235268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.234409 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:03:11.236858 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.236843 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:03:11.238053 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.238035 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:03:11.254040 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.254020 2570 log.go:25] "Validated CRI v1 runtime API" Jul 10 18:03:11.256325 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.256306 2570 log.go:25] "Validated CRI v1 image API" Jul 10 18:03:11.257224 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.257211 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 18:03:11.259800 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.259781 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a6e09f1-59ee-4f8f-8a2e-14dbf9c64259:/dev/nvme0n1p4 c3cf6a0d-2e20-4eb2-8f07-ad11d4c11b47:/dev/nvme0n1p3] Jul 10 18:03:11.259871 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.259800 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 18:03:11.261989 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.261973 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:03:11.267823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.267708 2570 manager.go:216] Machine: {Timestamp:2026-07-10 18:03:11.2654234 +0000 UTC m=+0.360613881 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098849 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb37b2231366f6cd894d797a59a21 SystemUUID:ec2bb37b-2231-366f-6cd8-94d797a59a21 BootID:47312c9c-0bb1-43e7-ab96-108d090c4962 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:75:14:c2:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:75:14:c2:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:63:53:88:54:6d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 18:03:11.267823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.267816 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 18:03:11.267944 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.267906 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 18:03:11.271409 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.271380 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 18:03:11.271554 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.271411 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-217.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 18:03:11.272311 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.272300 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 18:03:11.272351 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.272314 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 18:03:11.272351 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.272322 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:03:11.273236 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.273225 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:03:11.274753 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.274743 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 18:03:11.278424 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.278408 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:03:11.278558 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.278549 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 18:03:11.278591 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.278578 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 10 18:03:11.278629 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.278592 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 18:03:11.278629 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.278614 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 18:03:11.278629 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.278628 2570 kubelet.go:398] "Adding apiserver pod source" Jul 10 18:03:11.278731 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.278637 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 18:03:11.279220 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.279202 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:03:11.279313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.279226 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:03:11.282807 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.282785 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 18:03:11.283945 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.283918 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-77bld" Jul 10 18:03:11.287382 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287366 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 18:03:11.287450 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287397 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 18:03:11.287579 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287568 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 18:03:11.287646 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287585 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 18:03:11.287646 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287592 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 18:03:11.287646 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287617 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 18:03:11.287646 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287626 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 18:03:11.287646 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.287631 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 18:03:11.288240 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.288227 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 18:03:11.288290 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.288242 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 18:03:11.288290 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.288253 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 18:03:11.288290 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.288262 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 18:03:11.288290 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.288273 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 18:03:11.288290 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.288284 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 18:03:11.289297 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.289283 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 18:03:11.289297 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.289295 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 18:03:11.290262 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.290241 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 18:03:11.290344 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.290325 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 18:03:11.291749 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.291732 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-77bld" Jul 10 18:03:11.292852 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.292840 2570 server.go:1268] "Started kubelet" Jul 10 18:03:11.292957 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.292929 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 18:03:11.293014 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.292916 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 18:03:11.293014 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.292995 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 18:03:11.293814 ip-10-0-141-217 systemd[1]: Started Kubernetes Kubelet. Jul 10 18:03:11.294136 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.294117 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 10 18:03:11.294653 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.294568 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 18:03:11.299991 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.299966 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 18:03:11.300297 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.300280 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 18:03:11.300402 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.300304 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 18:03:11.300463 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.300391 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:11.301116 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301096 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 18:03:11.301116 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301100 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 18:03:11.301250 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301128 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 18:03:11.301250 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301200 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 10 18:03:11.301250 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301212 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 10 18:03:11.301353 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301293 2570 factory.go:55] Registering systemd factory Jul 10 18:03:11.301353 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301311 2570 factory.go:223] Registration of the systemd container factory successfully Jul 10 18:03:11.301503 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301493 2570 factory.go:153] Registering CRI-O factory Jul 10 18:03:11.301536 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301505 2570 factory.go:223] Registration of the crio container factory successfully Jul 10 18:03:11.301567 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301545 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 18:03:11.301567 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301564 2570 factory.go:103] Registering Raw factory Jul 10 18:03:11.301647 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301576 2570 manager.go:1196] Started watching for new ooms in manager Jul 10 18:03:11.301945 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.301936 2570 manager.go:349] Starting recovery of all containers Jul 10 18:03:11.302875 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.302854 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:03:11.306108 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.306083 2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-217.ec2.internal\" not found" node="ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.308151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.308129 2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-217.ec2.internal" not found Jul 10 18:03:11.312862 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.312841 2570 manager.go:354] Recovery completed Jul 10 18:03:11.319102 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.319089 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:03:11.321054 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.321037 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:03:11.321129 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.321074 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:03:11.321129 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.321089 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:03:11.321655 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.321641 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 18:03:11.321655 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.321654 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 18:03:11.321755 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.321676 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:03:11.321755 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.321656 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:03:11.322393 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.322379 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:03:11.322447 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.322407 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:03:11.322447 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.322417 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:03:11.323373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.323360 2570 policy_none.go:49] "None policy: Start" Jul 10 18:03:11.323420 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.323377 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 18:03:11.323420 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.323389 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 18:03:11.324245 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.324230 2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-217.ec2.internal" not found Jul 10 18:03:11.324932 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.324918 2570 policy_none.go:47] "Start" Jul 10 18:03:11.385373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.385217 2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-217.ec2.internal" not found Jul 10 18:03:11.388920 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.388906 2570 manager.go:337] "Starting Device Plugin manager" Jul 10 18:03:11.388992 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.388940 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 18:03:11.388992 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.388950 2570 server.go:87] "Starting device plugin registration server" Jul 10 18:03:11.389264 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.389252 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 18:03:11.389308 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.389266 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 18:03:11.389373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.389356 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 18:03:11.389473 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.389459 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 18:03:11.389533 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.389474 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 18:03:11.390134 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.390115 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 18:03:11.390208 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.390164 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:11.455946 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.455862 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 18:03:11.457233 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.457209 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 18:03:11.457334 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.457255 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 18:03:11.457334 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.457284 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 18:03:11.457334 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.457317 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 18:03:11.459927 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.459900 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:03:11.489778 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.489737 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:03:11.490719 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.490702 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:03:11.490793 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.490734 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:03:11.490793 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.490746 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:03:11.490793 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.490769 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.498318 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.498303 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.498368 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.498328 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-217.ec2.internal\": node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:11.510833 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.510810 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:11.558037 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.558009 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal"] Jul 10 18:03:11.558108 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.558081 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:03:11.558993 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.558973 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:03:11.559060 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.559005 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:03:11.559060 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.559015 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:03:11.560455 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.560443 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:03:11.560566 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.560552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.560613 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.560583 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:03:11.561151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.561128 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:03:11.561247 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.561164 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:03:11.561247 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.561174 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:03:11.561247 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.561132 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:03:11.561247 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.561236 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:03:11.561388 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.561249 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:03:11.562499 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.562485 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.562541 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.562510 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:03:11.563251 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.563238 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:03:11.563309 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.563263 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:03:11.563309 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.563276 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:03:11.578161 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.578135 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-217.ec2.internal\" not found" node="ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.582455 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.582441 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-217.ec2.internal\" not found" node="ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.602138 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.602116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8e066c4ae494a77030d5850328a24197-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal\" (UID: \"8e066c4ae494a77030d5850328a24197\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.611079 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.611059 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:11.702286 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.702254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8e066c4ae494a77030d5850328a24197-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal\" (UID: \"8e066c4ae494a77030d5850328a24197\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.702286 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.702287 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e066c4ae494a77030d5850328a24197-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal\" (UID: \"8e066c4ae494a77030d5850328a24197\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.702452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.702308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb22374ceca0abc340a036c543248c78-config\") pod \"kube-apiserver-proxy-ip-10-0-141-217.ec2.internal\" (UID: \"fb22374ceca0abc340a036c543248c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.702452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.702367 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8e066c4ae494a77030d5850328a24197-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal\" (UID: \"8e066c4ae494a77030d5850328a24197\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.711869 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.711797 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:11.802896 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.802847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e066c4ae494a77030d5850328a24197-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal\" (UID: \"8e066c4ae494a77030d5850328a24197\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.802896 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.802898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb22374ceca0abc340a036c543248c78-config\") pod \"kube-apiserver-proxy-ip-10-0-141-217.ec2.internal\" (UID: \"fb22374ceca0abc340a036c543248c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.803088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.802928 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb22374ceca0abc340a036c543248c78-config\") pod \"kube-apiserver-proxy-ip-10-0-141-217.ec2.internal\" (UID: \"fb22374ceca0abc340a036c543248c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.803088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.802934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e066c4ae494a77030d5850328a24197-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal\" (UID: \"8e066c4ae494a77030d5850328a24197\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.811925 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.811898 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:11.880070 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.880040 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.886567 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:11.886549 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" Jul 10 18:03:11.912812 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:11.912790 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:12.013461 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.013365 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:12.113762 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.113716 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-217.ec2.internal\" not found" Jul 10 18:03:12.142740 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.142708 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:03:12.203421 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.203377 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" Jul 10 18:03:12.234340 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.234304 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 18:03:12.234340 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.234334 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:03:12.234596 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.234475 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:03:12.234596 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.234485 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:03:12.234596 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.234519 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:03:12.236747 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.236729 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" Jul 10 18:03:12.270023 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.269919 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:03:12.279178 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.279145 2570 apiserver.go:52] "Watching apiserver" Jul 10 18:03:12.287761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.287726 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 18:03:12.289001 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.288977 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-d57jd","openshift-image-registry/node-ca-hjstd","openshift-multus/multus-sjwzz","openshift-network-diagnostics/network-check-target-kx74q","openshift-network-operator/iptables-alerter-xpsbq","openshift-ovn-kubernetes/ovnkube-node-6675h","kube-system/global-pull-secret-syncer-cl6p6","kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds","openshift-dns/node-resolver-zt5vq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal","openshift-multus/multus-additional-cni-plugins-wwslj","openshift-multus/network-metrics-daemon-mgjzs","kube-system/konnectivity-agent-xp5fp"] Jul 10 18:03:12.290630 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.290613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.290708 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.290688 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:12.291939 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.291915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:12.292038 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.292009 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:12.294761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.294732 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.296077 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.296023 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 17:58:12 +0000 UTC" deadline="2028-04-09 10:00:24.457133656 +0000 UTC" Jul 10 18:03:12.296077 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.296077 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15327h57m12.161061561s" Jul 10 18:03:12.297735 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.297709 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:12.297854 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.297805 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:12.297854 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.297833 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s45xp\"" Jul 10 18:03:12.298731 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.298703 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 18:03:12.298848 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.298732 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 18:03:12.298848 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.298769 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 18:03:12.298946 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.298921 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 18:03:12.299094 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.299057 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.300496 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.300475 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 18:03:12.300606 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.300516 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.301607 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.301587 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n8p2z\"" Jul 10 18:03:12.301739 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.301653 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 18:03:12.301739 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.301667 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:03:12.302003 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.301981 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 18:03:12.302151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.302136 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.303311 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.303295 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 18:03:12.303418 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.303327 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 18:03:12.304433 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.304415 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.305238 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305215 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 18:03:12.305328 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305316 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jm7pt\"" Jul 10 18:03:12.305392 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305382 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 18:03:12.305448 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305421 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 18:03:12.305448 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305430 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 18:03:12.305547 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305403 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 18:03:12.305906 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305887 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 18:03:12.305906 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305903 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dwl5b\"" Jul 10 18:03:12.306034 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:12.306034 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-cnibin\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306034 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2774c\" (UniqueName: \"kubernetes.io/projected/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-kube-api-access-2774c\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306034 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.305979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-node-log\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.306202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306036 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-log-socket\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.306202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-cni-bin\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.306202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.306202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.306202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ng8\" (UniqueName: \"kubernetes.io/projected/974c1314-41cd-413c-adf4-0ca46ac529f4-kube-api-access-d7ng8\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.306202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-socket-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.306202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-registration-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-daemon-config\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306247 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbed419f-4b6e-459c-a116-5c5ab7136daa-host-slash\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306264 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306280 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-system-cni-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306294 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-cni-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-hostroot\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smd8l\" (UniqueName: \"kubernetes.io/projected/dbed419f-4b6e-459c-a116-5c5ab7136daa-kube-api-access-smd8l\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-kubelet-config\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306354 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-dbus\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrkjj\" (UniqueName: \"kubernetes.io/projected/c04d6277-39ba-474e-af1e-3f61c91741c6-kube-api-access-lrkjj\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-socket-dir-parent\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-cni-bin\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-cni-multus\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306454 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-kubelet\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.306491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306469 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-systemd\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306514 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-os-release\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306550 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-cni-binary-copy\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306576 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-systemd-units\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-run-netns\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-var-lib-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-etc-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306677 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306696 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-ovn\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306729 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-etc-kubernetes\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306757 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-slash\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306780 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306806 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/974c1314-41cd-413c-adf4-0ca46ac529f4-ovn-node-metrics-cert\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-ovnkube-script-lib\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306857 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-device-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306905 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-etc-selinux\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflr9\" (UniqueName: \"kubernetes.io/projected/30a02ca9-8595-4ec2-a753-da9dceec7b49-kube-api-access-zflr9\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.307208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbed419f-4b6e-459c-a116-5c5ab7136daa-iptables-alerter-script\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.306997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-k8s-cni-cncf-io\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-kubelet\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-conf-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-ovnkube-config\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307158 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-env-overrides\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-multus-certs\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307188 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-netns\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307213 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-cni-netd\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307239 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-sys-fs\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307317 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kbqb8\"" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307369 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:03:12.307887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.307440 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 18:03:12.308528 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.308513 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.309002 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.308986 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-68kwh\"" Jul 10 18:03:12.309296 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.309281 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 18:03:12.309392 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.309375 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 18:03:12.309745 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.309730 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 18:03:12.310258 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.310243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.311021 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.311003 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn24c\"" Jul 10 18:03:12.311155 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.311139 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 18:03:12.311726 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.311711 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 18:03:12.312121 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.312097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.312867 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.312851 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 18:03:12.312867 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.312862 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 18:03:12.313176 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.313160 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sj2zt\"" Jul 10 18:03:12.314667 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.314649 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 18:03:12.314774 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.314759 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dz8t5\"" Jul 10 18:03:12.314877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.314861 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 18:03:12.319964 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.319941 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:03:12.340409 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.340380 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bk298" Jul 10 18:03:12.348827 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.348797 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bk298" Jul 10 18:03:12.358314 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.358283 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:03:12.402741 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.402706 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 18:03:12.407526 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-kubernetes\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.407526 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-host\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d407443-3c8b-4af7-8e0e-537bc17cdf8e-agent-certs\") pod \"konnectivity-agent-xp5fp\" (UID: \"7d407443-3c8b-4af7-8e0e-537bc17cdf8e\") " pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbed419f-4b6e-459c-a116-5c5ab7136daa-host-slash\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-system-cni-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-hostroot\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbed419f-4b6e-459c-a116-5c5ab7136daa-host-slash\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407696 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smd8l\" (UniqueName: \"kubernetes.io/projected/dbed419f-4b6e-459c-a116-5c5ab7136daa-kube-api-access-smd8l\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-hostroot\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407722 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/974c1314-41cd-413c-adf4-0ca46ac529f4-ovn-node-metrics-cert\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407754 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-system-cni-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.407789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-ovnkube-script-lib\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysctl-d\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-dbus\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-cni-multus\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407908 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-kubelet\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-var-lib-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-etc-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.407982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-ovn\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408007 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-host\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-os-release\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-cni-binary-copy\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408070 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-dbus\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408091 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-var-lib-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408087 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-run-netns\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408102 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-etc-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408134 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-run-netns\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408146 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-cni-multus\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.408313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408134 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408161 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-kubelet\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408196 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-ovn\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-etc-selinux\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408204 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-os-release\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysconfig\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-run\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408272 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-tuned\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408300 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-etc-selinux\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-cnibin\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbed419f-4b6e-459c-a116-5c5ab7136daa-iptables-alerter-script\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-k8s-cni-cncf-io\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408428 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-kubelet\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408430 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-ovnkube-script-lib\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408490 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-systemd\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408507 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-k8s-cni-cncf-io\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.409335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87485fb7-fa5b-4e65-af35-2df2664a7193-tmp\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbcb7ac8-3470-433d-9765-9cda8d1909f9-hosts-file\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408553 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-multus-certs\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408593 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-kubelet\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-netns\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408633 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408661 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-cni-binary-copy\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408662 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-sys\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-netns\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408725 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-run-multus-certs\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-lib-modules\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p85st\" (UniqueName: \"kubernetes.io/projected/dbcb7ac8-3470-433d-9765-9cda8d1909f9-kube-api-access-p85st\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408868 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-cni-bin\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-registration-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408922 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d407443-3c8b-4af7-8e0e-537bc17cdf8e-konnectivity-ca\") pod \"konnectivity-agent-xp5fp\" (UID: \"7d407443-3c8b-4af7-8e0e-537bc17cdf8e\") " pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.410588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408942 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-cni-bin\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-daemon-config\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.408957 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.408986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409006 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbed419f-4b6e-459c-a116-5c5ab7136daa-iptables-alerter-script\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.409038 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs podName:c04d6277-39ba-474e-af1e-3f61c91741c6 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:12.909015945 +0000 UTC m=+2.004206426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs") pod "network-metrics-daemon-mgjzs" (UID: "c04d6277-39ba-474e-af1e-3f61c91741c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-cni-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.409085 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflr9\" (UniqueName: \"kubernetes.io/projected/30a02ca9-8595-4ec2-a753-da9dceec7b49-kube-api-access-zflr9\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-kubelet-config\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrkjj\" (UniqueName: \"kubernetes.io/projected/c04d6277-39ba-474e-af1e-3f61c91741c6-kube-api-access-lrkjj\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-socket-dir-parent\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-cni-bin\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-systemd\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-ovnkube-config\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.411153 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.409275 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret podName:ce68a2f5-08f1-47cf-80c8-004a8ac733bb nodeName:}" failed. No retries permitted until 2026-07-10 18:03:12.909251525 +0000 UTC m=+2.004441996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret") pod "global-pull-secret-syncer-cl6p6" (UID: "ce68a2f5-08f1-47cf-80c8-004a8ac733bb") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409354 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-cni-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-socket-dir-parent\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-env-overrides\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409463 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-daemon-config\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409529 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-kubelet-config\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409532 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-host-var-lib-cni-bin\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-systemd-units\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409587 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-systemd\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409646 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-registration-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-systemd-units\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-modprobe-d\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746n8\" (UniqueName: \"kubernetes.io/projected/87485fb7-fa5b-4e65-af35-2df2664a7193-kube-api-access-746n8\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-serviceca\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409824 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcb7ac8-3470-433d-9765-9cda8d1909f9-tmp-dir\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.411632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-etc-kubernetes\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-slash\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-etc-kubernetes\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-device-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409965 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-conf-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409984 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-slash\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-env-overrides\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.409993 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysctl-conf\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-var-lib-kubelet\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410030 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-run-openvswitch\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410030 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/974c1314-41cd-413c-adf4-0ca46ac529f4-ovnkube-config\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-cni-netd\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410052 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-multus-conf-dir\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410076 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-device-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-sys-fs\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-cni-netd\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-os-release\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410127 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-sys-fs\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-cnibin\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2774c\" (UniqueName: \"kubernetes.io/projected/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-kube-api-access-2774c\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410210 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-node-log\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410221 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-cnibin\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-log-socket\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410268 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-node-log\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410313 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/974c1314-41cd-413c-adf4-0ca46ac529f4-log-socket\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410351 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzx5\" (UniqueName: \"kubernetes.io/projected/32795b85-db8f-480d-9630-2d1d11a12bd3-kube-api-access-qvzx5\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcvj\" (UniqueName: \"kubernetes.io/projected/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-kube-api-access-9kcvj\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410408 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ng8\" (UniqueName: \"kubernetes.io/projected/974c1314-41cd-413c-adf4-0ca46ac529f4-kube-api-access-d7ng8\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-socket-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.412765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410512 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-system-cni-dir\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.413300 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.410616 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30a02ca9-8595-4ec2-a753-da9dceec7b49-socket-dir\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.413300 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.412299 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/974c1314-41cd-413c-adf4-0ca46ac529f4-ovn-node-metrics-cert\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.425287 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.425259 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:03:12.425287 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.425288 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:03:12.425486 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.425303 2570 projected.go:196] Error preparing data for projected volume kube-api-access-c9cjn for pod openshift-network-diagnostics/network-check-target-kx74q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:12.425486 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.425364 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn podName:7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:12.925346344 +0000 UTC m=+2.020536812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c9cjn" (UniqueName: "kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn") pod "network-check-target-kx74q" (UID: "7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:12.428017 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.427987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflr9\" (UniqueName: \"kubernetes.io/projected/30a02ca9-8595-4ec2-a753-da9dceec7b49-kube-api-access-zflr9\") pod \"aws-ebs-csi-driver-node-gvfds\" (UID: \"30a02ca9-8595-4ec2-a753-da9dceec7b49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.428148 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.428010 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smd8l\" (UniqueName: \"kubernetes.io/projected/dbed419f-4b6e-459c-a116-5c5ab7136daa-kube-api-access-smd8l\") pod \"iptables-alerter-xpsbq\" (UID: \"dbed419f-4b6e-459c-a116-5c5ab7136daa\") " pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.428211 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.428181 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2774c\" (UniqueName: \"kubernetes.io/projected/8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f-kube-api-access-2774c\") pod \"multus-sjwzz\" (UID: \"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f\") " pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.428450 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.428433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ng8\" (UniqueName: \"kubernetes.io/projected/974c1314-41cd-413c-adf4-0ca46ac529f4-kube-api-access-d7ng8\") pod \"ovnkube-node-6675h\" (UID: \"974c1314-41cd-413c-adf4-0ca46ac529f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.429362 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.429343 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrkjj\" (UniqueName: \"kubernetes.io/projected/c04d6277-39ba-474e-af1e-3f61c91741c6-kube-api-access-lrkjj\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:12.467752 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.467722 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb22374ceca0abc340a036c543248c78.slice/crio-e6ebb82f53e747a0b06e49317f14ebb91130440dcbc995771b61caf7003887bf WatchSource:0}: Error finding container e6ebb82f53e747a0b06e49317f14ebb91130440dcbc995771b61caf7003887bf: Status 404 returned error can't find the container with id e6ebb82f53e747a0b06e49317f14ebb91130440dcbc995771b61caf7003887bf Jul 10 18:03:12.469617 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.469053 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e066c4ae494a77030d5850328a24197.slice/crio-388e13055f4cce271bde13dc7b4f95183c3e60d7e25121fcfd6853100bc69b07 WatchSource:0}: Error finding container 388e13055f4cce271bde13dc7b4f95183c3e60d7e25121fcfd6853100bc69b07: Status 404 returned error can't find the container with id 388e13055f4cce271bde13dc7b4f95183c3e60d7e25121fcfd6853100bc69b07 Jul 10 18:03:12.476235 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.476216 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:03:12.510933 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.510899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-os-release\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.510933 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.510938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzx5\" (UniqueName: \"kubernetes.io/projected/32795b85-db8f-480d-9630-2d1d11a12bd3-kube-api-access-qvzx5\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.510959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcvj\" (UniqueName: \"kubernetes.io/projected/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-kube-api-access-9kcvj\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.510978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-system-cni-dir\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-kubernetes\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-host\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d407443-3c8b-4af7-8e0e-537bc17cdf8e-agent-certs\") pod \"konnectivity-agent-xp5fp\" (UID: \"7d407443-3c8b-4af7-8e0e-537bc17cdf8e\") " pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511064 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-os-release\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysctl-d\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511123 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-host\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-kubernetes\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511084 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-system-cni-dir\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-host\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysconfig\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511213 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511220 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-run\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-tuned\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511240 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-host\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511272 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-cnibin\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511287 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysctl-d\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-systemd\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511320 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-run\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87485fb7-fa5b-4e65-af35-2df2664a7193-tmp\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysconfig\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-cnibin\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511383 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-systemd\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbcb7ac8-3470-433d-9765-9cda8d1909f9-hosts-file\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511446 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511475 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-sys\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511502 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-lib-modules\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511524 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbcb7ac8-3470-433d-9765-9cda8d1909f9-hosts-file\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p85st\" (UniqueName: \"kubernetes.io/projected/dbcb7ac8-3470-433d-9765-9cda8d1909f9-kube-api-access-p85st\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d407443-3c8b-4af7-8e0e-537bc17cdf8e-konnectivity-ca\") pod \"konnectivity-agent-xp5fp\" (UID: \"7d407443-3c8b-4af7-8e0e-537bc17cdf8e\") " pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.511972 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-modprobe-d\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511762 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-sys\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746n8\" (UniqueName: \"kubernetes.io/projected/87485fb7-fa5b-4e65-af35-2df2664a7193-kube-api-access-746n8\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511829 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-serviceca\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcb7ac8-3470-433d-9765-9cda8d1909f9-tmp-dir\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysctl-conf\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.511884 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-lib-modules\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512069 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32795b85-db8f-480d-9630-2d1d11a12bd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512110 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-var-lib-kubelet\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512188 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-var-lib-kubelet\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512459 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512577 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-modprobe-d\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.512877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.513405 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512892 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-serviceca\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.513405 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbcb7ac8-3470-433d-9765-9cda8d1909f9-tmp-dir\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.513405 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512896 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-sysctl-conf\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.513405 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.512961 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32795b85-db8f-480d-9630-2d1d11a12bd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.513405 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.513215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d407443-3c8b-4af7-8e0e-537bc17cdf8e-konnectivity-ca\") pod \"konnectivity-agent-xp5fp\" (UID: \"7d407443-3c8b-4af7-8e0e-537bc17cdf8e\") " pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.514427 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.514398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87485fb7-fa5b-4e65-af35-2df2664a7193-tmp\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.514566 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.514514 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87485fb7-fa5b-4e65-af35-2df2664a7193-etc-tuned\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.514812 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.514768 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d407443-3c8b-4af7-8e0e-537bc17cdf8e-agent-certs\") pod \"konnectivity-agent-xp5fp\" (UID: \"7d407443-3c8b-4af7-8e0e-537bc17cdf8e\") " pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.519585 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.519561 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcvj\" (UniqueName: \"kubernetes.io/projected/e8c2b25e-ffaf-48b9-b544-fff1ab93f1da-kube-api-access-9kcvj\") pod \"node-ca-hjstd\" (UID: \"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da\") " pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.519924 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.519903 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p85st\" (UniqueName: \"kubernetes.io/projected/dbcb7ac8-3470-433d-9765-9cda8d1909f9-kube-api-access-p85st\") pod \"node-resolver-zt5vq\" (UID: \"dbcb7ac8-3470-433d-9765-9cda8d1909f9\") " pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.520205 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.520147 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzx5\" (UniqueName: \"kubernetes.io/projected/32795b85-db8f-480d-9630-2d1d11a12bd3-kube-api-access-qvzx5\") pod \"multus-additional-cni-plugins-wwslj\" (UID: \"32795b85-db8f-480d-9630-2d1d11a12bd3\") " pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.520428 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.520407 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-746n8\" (UniqueName: \"kubernetes.io/projected/87485fb7-fa5b-4e65-af35-2df2664a7193-kube-api-access-746n8\") pod \"tuned-d57jd\" (UID: \"87485fb7-fa5b-4e65-af35-2df2664a7193\") " pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.635851 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.635797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sjwzz" Jul 10 18:03:12.645545 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.645515 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af2f4d1_f8be_47b7_b4e9_ed5d6d2be08f.slice/crio-765c0094a8cfa3b4230bbd644f205c22e4e79a2ae97e4620b6daf14931326641 WatchSource:0}: Error finding container 765c0094a8cfa3b4230bbd644f205c22e4e79a2ae97e4620b6daf14931326641: Status 404 returned error can't find the container with id 765c0094a8cfa3b4230bbd644f205c22e4e79a2ae97e4620b6daf14931326641 Jul 10 18:03:12.647226 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.647211 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xpsbq" Jul 10 18:03:12.654989 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.654961 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbed419f_4b6e_459c_a116_5c5ab7136daa.slice/crio-278d312e2fa1518f510b6651084271f2e26f75230de75dced3835f0840a0bf43 WatchSource:0}: Error finding container 278d312e2fa1518f510b6651084271f2e26f75230de75dced3835f0840a0bf43: Status 404 returned error can't find the container with id 278d312e2fa1518f510b6651084271f2e26f75230de75dced3835f0840a0bf43 Jul 10 18:03:12.659287 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.659268 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:12.666535 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.666505 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" Jul 10 18:03:12.667305 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.667284 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974c1314_41cd_413c_adf4_0ca46ac529f4.slice/crio-213b1eccfbfa05140d1c6774107a0fd2d16db1fa89e6178cd40dbd935bc21660 WatchSource:0}: Error finding container 213b1eccfbfa05140d1c6774107a0fd2d16db1fa89e6178cd40dbd935bc21660: Status 404 returned error can't find the container with id 213b1eccfbfa05140d1c6774107a0fd2d16db1fa89e6178cd40dbd935bc21660 Jul 10 18:03:12.675509 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.675467 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a02ca9_8595_4ec2_a753_da9dceec7b49.slice/crio-c1d33522ff622505c9dae2c16cd07da1ad58f97854f98904fde1a04329b85ba1 WatchSource:0}: Error finding container c1d33522ff622505c9dae2c16cd07da1ad58f97854f98904fde1a04329b85ba1: Status 404 returned error can't find the container with id c1d33522ff622505c9dae2c16cd07da1ad58f97854f98904fde1a04329b85ba1 Jul 10 18:03:12.685711 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.685693 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d57jd" Jul 10 18:03:12.692979 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.692958 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87485fb7_fa5b_4e65_af35_2df2664a7193.slice/crio-6038fe4104565e57a0485e6bdeada909760d847de635d6f7eda38e49f03bed7c WatchSource:0}: Error finding container 6038fe4104565e57a0485e6bdeada909760d847de635d6f7eda38e49f03bed7c: Status 404 returned error can't find the container with id 6038fe4104565e57a0485e6bdeada909760d847de635d6f7eda38e49f03bed7c Jul 10 18:03:12.706081 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.706050 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hjstd" Jul 10 18:03:12.714819 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.714787 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c2b25e_ffaf_48b9_b544_fff1ab93f1da.slice/crio-10872c1d429364d82626ca71369cb1816998b7fe9d5eb1b80d44c19ef14d786c WatchSource:0}: Error finding container 10872c1d429364d82626ca71369cb1816998b7fe9d5eb1b80d44c19ef14d786c: Status 404 returned error can't find the container with id 10872c1d429364d82626ca71369cb1816998b7fe9d5eb1b80d44c19ef14d786c Jul 10 18:03:12.722561 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.722542 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zt5vq" Jul 10 18:03:12.731055 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.731033 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcb7ac8_3470_433d_9765_9cda8d1909f9.slice/crio-348604d48daf65b7cc9665c82cecdafe1bb79c0e43c41e1a18441380b2190f52 WatchSource:0}: Error finding container 348604d48daf65b7cc9665c82cecdafe1bb79c0e43c41e1a18441380b2190f52: Status 404 returned error can't find the container with id 348604d48daf65b7cc9665c82cecdafe1bb79c0e43c41e1a18441380b2190f52 Jul 10 18:03:12.747608 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.747586 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:12.754505 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.754484 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wwslj" Jul 10 18:03:12.756171 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.756148 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d407443_3c8b_4af7_8e0e_537bc17cdf8e.slice/crio-a150e2ea5a74ec7fff24bce0024ccde3c6aac0a00c99059d60864bb7fea68c87 WatchSource:0}: Error finding container a150e2ea5a74ec7fff24bce0024ccde3c6aac0a00c99059d60864bb7fea68c87: Status 404 returned error can't find the container with id a150e2ea5a74ec7fff24bce0024ccde3c6aac0a00c99059d60864bb7fea68c87 Jul 10 18:03:12.762527 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:03:12.762505 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32795b85_db8f_480d_9630_2d1d11a12bd3.slice/crio-661e2742afcbb1ecd4cf18de8e1b30325998f22fe8f7a30b12f5ce2ec8bc67c0 WatchSource:0}: Error finding container 661e2742afcbb1ecd4cf18de8e1b30325998f22fe8f7a30b12f5ce2ec8bc67c0: Status 404 returned error can't find the container with id 661e2742afcbb1ecd4cf18de8e1b30325998f22fe8f7a30b12f5ce2ec8bc67c0 Jul 10 18:03:12.916074 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.915983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:12.916074 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:12.916032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:12.916256 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.916143 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:12.916256 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.916143 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:12.916256 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.916214 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret podName:ce68a2f5-08f1-47cf-80c8-004a8ac733bb nodeName:}" failed. No retries permitted until 2026-07-10 18:03:13.916197671 +0000 UTC m=+3.011388149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret") pod "global-pull-secret-syncer-cl6p6" (UID: "ce68a2f5-08f1-47cf-80c8-004a8ac733bb") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:12.916256 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:12.916231 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs podName:c04d6277-39ba-474e-af1e-3f61c91741c6 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:13.91622366 +0000 UTC m=+3.011414126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs") pod "network-metrics-daemon-mgjzs" (UID: "c04d6277-39ba-474e-af1e-3f61c91741c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:13.017249 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.017205 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:13.017533 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.017471 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:03:13.017533 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.017502 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:03:13.017533 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.017518 2570 projected.go:196] Error preparing data for projected volume kube-api-access-c9cjn for pod openshift-network-diagnostics/network-check-target-kx74q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:13.017765 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.017588 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn podName:7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:14.017565018 +0000 UTC m=+3.112755505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9cjn" (UniqueName: "kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn") pod "network-check-target-kx74q" (UID: "7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:13.110076 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.110041 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:03:13.351752 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.351638 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:58:13 +0000 UTC" deadline="2028-03-17 00:02:12.222062302 +0000 UTC" Jul 10 18:03:13.351752 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.351702 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14765h58m58.870364837s" Jul 10 18:03:13.457830 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.457795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:13.458024 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.457958 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:13.458504 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.458468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:13.458719 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.458627 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:13.521525 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.521440 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sjwzz" event={"ID":"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f","Type":"ContainerStarted","Data":"765c0094a8cfa3b4230bbd644f205c22e4e79a2ae97e4620b6daf14931326641"} Jul 10 18:03:13.532807 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.532751 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" event={"ID":"fb22374ceca0abc340a036c543248c78","Type":"ContainerStarted","Data":"e6ebb82f53e747a0b06e49317f14ebb91130440dcbc995771b61caf7003887bf"} Jul 10 18:03:13.563926 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.563812 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" event={"ID":"8e066c4ae494a77030d5850328a24197","Type":"ContainerStarted","Data":"388e13055f4cce271bde13dc7b4f95183c3e60d7e25121fcfd6853100bc69b07"} Jul 10 18:03:13.590689 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.590625 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xp5fp" event={"ID":"7d407443-3c8b-4af7-8e0e-537bc17cdf8e","Type":"ContainerStarted","Data":"a150e2ea5a74ec7fff24bce0024ccde3c6aac0a00c99059d60864bb7fea68c87"} Jul 10 18:03:13.604406 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.604309 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zt5vq" event={"ID":"dbcb7ac8-3470-433d-9765-9cda8d1909f9","Type":"ContainerStarted","Data":"348604d48daf65b7cc9665c82cecdafe1bb79c0e43c41e1a18441380b2190f52"} Jul 10 18:03:13.610216 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.610014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"213b1eccfbfa05140d1c6774107a0fd2d16db1fa89e6178cd40dbd935bc21660"} Jul 10 18:03:13.626822 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.626781 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xpsbq" event={"ID":"dbed419f-4b6e-459c-a116-5c5ab7136daa","Type":"ContainerStarted","Data":"278d312e2fa1518f510b6651084271f2e26f75230de75dced3835f0840a0bf43"} Jul 10 18:03:13.632218 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.632175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerStarted","Data":"661e2742afcbb1ecd4cf18de8e1b30325998f22fe8f7a30b12f5ce2ec8bc67c0"} Jul 10 18:03:13.648349 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.648261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hjstd" event={"ID":"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da","Type":"ContainerStarted","Data":"10872c1d429364d82626ca71369cb1816998b7fe9d5eb1b80d44c19ef14d786c"} Jul 10 18:03:13.669207 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.669169 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d57jd" event={"ID":"87485fb7-fa5b-4e65-af35-2df2664a7193","Type":"ContainerStarted","Data":"6038fe4104565e57a0485e6bdeada909760d847de635d6f7eda38e49f03bed7c"} Jul 10 18:03:13.684701 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.684645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" event={"ID":"30a02ca9-8595-4ec2-a753-da9dceec7b49","Type":"ContainerStarted","Data":"c1d33522ff622505c9dae2c16cd07da1ad58f97854f98904fde1a04329b85ba1"} Jul 10 18:03:13.774985 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.774948 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:03:13.926713 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.926674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:13.926893 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:13.926747 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:13.926955 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.926915 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:13.927021 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.926983 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret podName:ce68a2f5-08f1-47cf-80c8-004a8ac733bb nodeName:}" failed. No retries permitted until 2026-07-10 18:03:15.926962578 +0000 UTC m=+5.022153050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret") pod "global-pull-secret-syncer-cl6p6" (UID: "ce68a2f5-08f1-47cf-80c8-004a8ac733bb") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:13.927447 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.927413 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:13.927541 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:13.927471 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs podName:c04d6277-39ba-474e-af1e-3f61c91741c6 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:15.927452683 +0000 UTC m=+5.022643155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs") pod "network-metrics-daemon-mgjzs" (UID: "c04d6277-39ba-474e-af1e-3f61c91741c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:14.027155 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:14.027120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:14.027336 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:14.027281 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:03:14.027336 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:14.027301 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:03:14.027336 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:14.027314 2570 projected.go:196] Error preparing data for projected volume kube-api-access-c9cjn for pod openshift-network-diagnostics/network-check-target-kx74q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:14.027520 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:14.027360 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn podName:7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:16.027346026 +0000 UTC m=+5.122536506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9cjn" (UniqueName: "kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn") pod "network-check-target-kx74q" (UID: "7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:14.313888 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:14.313829 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:03:14.352549 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:14.352479 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:58:13 +0000 UTC" deadline="2028-02-15 11:14:54.12886656 +0000 UTC" Jul 10 18:03:14.352549 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:14.352546 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14033h11m39.77632533s" Jul 10 18:03:14.458430 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:14.458394 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:14.458635 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:14.458552 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:15.458548 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:15.458479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:15.459133 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:15.458643 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:15.459133 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:15.459004 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:15.459133 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:15.459105 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:15.948877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:15.948745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:15.948877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:15.948817 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:15.949146 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:15.948968 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:15.949146 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:15.949036 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret podName:ce68a2f5-08f1-47cf-80c8-004a8ac733bb nodeName:}" failed. No retries permitted until 2026-07-10 18:03:19.949013744 +0000 UTC m=+9.044204217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret") pod "global-pull-secret-syncer-cl6p6" (UID: "ce68a2f5-08f1-47cf-80c8-004a8ac733bb") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:15.949146 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:15.949112 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:15.949146 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:15.949148 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs podName:c04d6277-39ba-474e-af1e-3f61c91741c6 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:19.949135292 +0000 UTC m=+9.044325763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs") pod "network-metrics-daemon-mgjzs" (UID: "c04d6277-39ba-474e-af1e-3f61c91741c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:16.049592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:16.049553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:16.049864 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:16.049799 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:03:16.049864 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:16.049822 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:03:16.049864 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:16.049835 2570 projected.go:196] Error preparing data for projected volume kube-api-access-c9cjn for pod openshift-network-diagnostics/network-check-target-kx74q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:16.050038 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:16.049893 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn podName:7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:20.049873568 +0000 UTC m=+9.145064044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9cjn" (UniqueName: "kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn") pod "network-check-target-kx74q" (UID: "7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:16.457910 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:16.457873 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:16.458099 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:16.458038 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:17.457733 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:17.457695 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:17.457733 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:17.457732 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:17.458259 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:17.457950 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:17.458259 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:17.458059 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:18.457615 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:18.457556 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:18.457833 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:18.457733 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:19.458894 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:19.458799 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:19.459367 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:19.458939 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:19.460998 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:19.460842 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:19.460998 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:19.460949 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:19.985494 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:19.985457 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:19.985716 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:19.985517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:19.985716 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:19.985671 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:19.985716 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:19.985673 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:19.985910 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:19.985742 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret podName:ce68a2f5-08f1-47cf-80c8-004a8ac733bb nodeName:}" failed. No retries permitted until 2026-07-10 18:03:27.985723047 +0000 UTC m=+17.080913531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret") pod "global-pull-secret-syncer-cl6p6" (UID: "ce68a2f5-08f1-47cf-80c8-004a8ac733bb") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:19.985910 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:19.985763 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs podName:c04d6277-39ba-474e-af1e-3f61c91741c6 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:27.985753331 +0000 UTC m=+17.080943801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs") pod "network-metrics-daemon-mgjzs" (UID: "c04d6277-39ba-474e-af1e-3f61c91741c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:20.086265 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:20.086226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:20.086461 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:20.086441 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:03:20.086542 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:20.086465 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:03:20.086542 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:20.086478 2570 projected.go:196] Error preparing data for projected volume kube-api-access-c9cjn for pod openshift-network-diagnostics/network-check-target-kx74q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:20.086542 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:20.086541 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn podName:7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:28.086522378 +0000 UTC m=+17.181712859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9cjn" (UniqueName: "kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn") pod "network-check-target-kx74q" (UID: "7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:20.459223 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:20.458586 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:20.459223 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:20.458756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:21.459989 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:21.459340 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:21.459989 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:21.459454 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:21.459989 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:21.459845 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:21.459989 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:21.459944 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:22.458316 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:22.458277 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:22.458490 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:22.458426 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:23.458251 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:23.457722 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:23.458251 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:23.457766 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:23.458251 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:23.457862 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:23.458251 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:23.458019 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:24.457679 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:24.457637 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:24.457843 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:24.457810 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:25.458592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:25.458553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:25.459021 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:25.458553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:25.459021 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:25.458687 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:25.459021 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:25.458763 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:26.458544 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:26.458502 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:26.458740 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:26.458674 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:27.457790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:27.457750 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:27.457957 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:27.457863 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:27.457957 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:27.457925 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:27.458085 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:27.458020 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:28.053334 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:28.053303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:28.053809 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:28.053346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:28.053809 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.053460 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:28.053809 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.053470 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:28.053809 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.053514 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret podName:ce68a2f5-08f1-47cf-80c8-004a8ac733bb nodeName:}" failed. No retries permitted until 2026-07-10 18:03:44.053500209 +0000 UTC m=+33.148690672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret") pod "global-pull-secret-syncer-cl6p6" (UID: "ce68a2f5-08f1-47cf-80c8-004a8ac733bb") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:28.053809 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.053537 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs podName:c04d6277-39ba-474e-af1e-3f61c91741c6 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:44.053519437 +0000 UTC m=+33.148709908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs") pod "network-metrics-daemon-mgjzs" (UID: "c04d6277-39ba-474e-af1e-3f61c91741c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:28.154690 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:28.154655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:28.154849 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.154812 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:03:28.154849 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.154831 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:03:28.154849 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.154843 2570 projected.go:196] Error preparing data for projected volume kube-api-access-c9cjn for pod openshift-network-diagnostics/network-check-target-kx74q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:28.154992 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.154892 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn podName:7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2 nodeName:}" failed. No retries permitted until 2026-07-10 18:03:44.154879115 +0000 UTC m=+33.250069579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9cjn" (UniqueName: "kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn") pod "network-check-target-kx74q" (UID: "7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:28.458237 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:28.458153 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:28.458399 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:28.458293 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:29.457903 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:29.457857 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:29.457903 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:29.457890 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:29.458341 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:29.457980 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:29.458341 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:29.458103 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:30.458000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:30.457962 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:30.458380 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:30.458087 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:31.458823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:31.458782 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:31.459300 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:31.458831 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:31.459300 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:31.458902 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:31.459300 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:31.459029 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:32.457614 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:32.457568 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:32.457802 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:32.457732 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:33.457695 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:33.457656 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:33.458250 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:33.457668 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:33.458250 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:33.457783 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:33.458250 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:33.457897 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:34.458133 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.457927 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:34.458777 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:34.458207 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:34.739241 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739166 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:03:34.739483 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739463 2570 generic.go:356] "Generic (PLEG): container finished" podID="974c1314-41cd-413c-adf4-0ca46ac529f4" containerID="f8d2625cb290debca2b0aa4913dab0322c2e00c07a1423c3e67da9b9c9b9b330" exitCode=1 Jul 10 18:03:34.739569 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739538 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"e8c84040f17f8e34499a2e33f26c219c3fbea2c937d8937d3579b220eea4af1a"} Jul 10 18:03:34.739655 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739637 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"e1313a83b314625c44003c39a555e80f7739ab4b8f837a56cedc32261fb7e85b"} Jul 10 18:03:34.739692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739684 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"3e86da53f9aeda4d0fa60200a0dacea8695a0cc872b412ef78903bd39fb62261"} Jul 10 18:03:34.739737 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"1dac40d73921d3f15f8a59ebfbd140cfe26e3763fb1cd08072fd8d3e8f99c779"} Jul 10 18:03:34.739794 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739777 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerDied","Data":"f8d2625cb290debca2b0aa4913dab0322c2e00c07a1423c3e67da9b9c9b9b330"} Jul 10 18:03:34.739907 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.739814 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"32a517fb73ed6aac70a84c331a7d7a0304bf8a8ca8cd4731fdfd832f0be4b41c"} Jul 10 18:03:34.741147 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.740989 2570 generic.go:356] "Generic (PLEG): container finished" podID="32795b85-db8f-480d-9630-2d1d11a12bd3" containerID="9c78ec0fdf37753b4d2d63010d48255819e37f769bf319e383812f9ce7e41f18" exitCode=0 Jul 10 18:03:34.741228 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.741019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerDied","Data":"9c78ec0fdf37753b4d2d63010d48255819e37f769bf319e383812f9ce7e41f18"} Jul 10 18:03:34.742712 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.742681 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hjstd" event={"ID":"e8c2b25e-ffaf-48b9-b544-fff1ab93f1da","Type":"ContainerStarted","Data":"c1ac48fb4c5ea15d82654e7fb2d4cf30d0ec0b689d27af1e85c91f9b4913902c"} Jul 10 18:03:34.744172 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.744138 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d57jd" event={"ID":"87485fb7-fa5b-4e65-af35-2df2664a7193","Type":"ContainerStarted","Data":"085f71122e7a36bc164936dcff638ff078efddb97018c30ab9cdd08e15f06dac"} Jul 10 18:03:34.745842 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.745817 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" event={"ID":"30a02ca9-8595-4ec2-a753-da9dceec7b49","Type":"ContainerStarted","Data":"7a9044f4a858eef29a443302446aed9d4b0e12a983bc78683a159aa5f1bf89af"} Jul 10 18:03:34.747540 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.747523 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sjwzz" event={"ID":"8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f","Type":"ContainerStarted","Data":"1aad1a712c6db790c050c0a2d87c1a2cc4ccb5ef6a5a83b49c4fac649ddec5ae"} Jul 10 18:03:34.748856 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.748831 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" event={"ID":"fb22374ceca0abc340a036c543248c78","Type":"ContainerStarted","Data":"6a3347aa93e6a158b01ed7218fa9d6d4a566412f1a40bc5aa547e76f5413deac"} Jul 10 18:03:34.750151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.750131 2570 generic.go:356] "Generic (PLEG): container finished" podID="8e066c4ae494a77030d5850328a24197" containerID="0f17e6b3a17158b4ebf2edb2405b0e2a567f402e2ad642f3bfec727d0e28956e" exitCode=0 Jul 10 18:03:34.750244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.750207 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" event={"ID":"8e066c4ae494a77030d5850328a24197","Type":"ContainerDied","Data":"0f17e6b3a17158b4ebf2edb2405b0e2a567f402e2ad642f3bfec727d0e28956e"} Jul 10 18:03:34.751569 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.751546 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xp5fp" event={"ID":"7d407443-3c8b-4af7-8e0e-537bc17cdf8e","Type":"ContainerStarted","Data":"e01c92acaafddd05ff33a09d420fbb6de6c7b4c47d8173ec6ef008f0e44e9d16"} Jul 10 18:03:34.752908 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.752890 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zt5vq" event={"ID":"dbcb7ac8-3470-433d-9765-9cda8d1909f9","Type":"ContainerStarted","Data":"65adee009bed6721ff9e0c049ab99f11eeed082f06dc192481a7adc988db3c21"} Jul 10 18:03:34.782546 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.782499 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sjwzz" podStartSLOduration=1.5891769789999999 podStartE2EDuration="22.782481754s" totalImagesPullingTime="21.193304775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:34.781872073 +0000 UTC m=+23.877062755" watchObservedRunningTime="2026-07-10 18:03:34.782481754 +0000 UTC m=+23.877672241" Jul 10 18:03:34.814740 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.814699 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d57jd" podStartSLOduration=1.74332452 podStartE2EDuration="22.814688985s" totalImagesPullingTime="21.071364465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:34.800075147 +0000 UTC m=+23.895265632" watchObservedRunningTime="2026-07-10 18:03:34.814688985 +0000 UTC m=+23.909879471" Jul 10 18:03:34.829083 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.829048 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hjstd" podStartSLOduration=1.783121548 podStartE2EDuration="22.829033762s" totalImagesPullingTime="21.045912214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:34.828853055 +0000 UTC m=+23.924043540" watchObservedRunningTime="2026-07-10 18:03:34.829033762 +0000 UTC m=+23.924224246" Jul 10 18:03:34.861736 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.861686 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zt5vq" podStartSLOduration=1.8235853180000001 podStartE2EDuration="22.861669283s" totalImagesPullingTime="21.038083965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:34.845557253 +0000 UTC m=+23.940747738" watchObservedRunningTime="2026-07-10 18:03:34.861669283 +0000 UTC m=+23.956859843" Jul 10 18:03:34.861960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.861932 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xp5fp" podStartSLOduration=1.855818427 podStartE2EDuration="22.861922704s" totalImagesPullingTime="21.006104277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:34.861111888 +0000 UTC m=+23.956302374" watchObservedRunningTime="2026-07-10 18:03:34.861922704 +0000 UTC m=+23.957113190" Jul 10 18:03:34.875274 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:34.875234 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-217.ec2.internal" podStartSLOduration=21.87521989 podStartE2EDuration="21.87521989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:34.874979502 +0000 UTC m=+23.970169987" watchObservedRunningTime="2026-07-10 18:03:34.87521989 +0000 UTC m=+23.970410375" Jul 10 18:03:35.458302 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:35.458233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:35.458848 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:35.458364 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:35.458848 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:35.458408 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:35.458848 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:35.458496 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:35.756808 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:35.756765 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xpsbq" event={"ID":"dbed419f-4b6e-459c-a116-5c5ab7136daa","Type":"ContainerStarted","Data":"b82201ed1c953b818ecccc3de0ad05b71836bb6a80aa285050efcdc8eceeee9c"} Jul 10 18:03:35.758550 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:35.758522 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" event={"ID":"8e066c4ae494a77030d5850328a24197","Type":"ContainerStarted","Data":"ec465d093e7ca0660311558e68aa5f28c13d1dcf9240338a7cc0ff10b59468ce"} Jul 10 18:03:35.795773 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:35.795728 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-217.ec2.internal" podStartSLOduration=22.795710523 podStartE2EDuration="22.795710523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:35.780460898 +0000 UTC m=+24.875651383" watchObservedRunningTime="2026-07-10 18:03:35.795710523 +0000 UTC m=+24.890901009" Jul 10 18:03:35.876372 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:35.876347 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 18:03:36.410356 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:36.404575 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:03:35.876367054Z","UUID":"134e8bc1-3de1-4ecc-9eda-cb27c73bd843","Handler":null,"Name":"","Endpoint":""} Jul 10 18:03:36.412336 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:36.412309 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 18:03:36.412499 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:36.412354 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 18:03:36.457536 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:36.457509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:36.457705 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:36.457650 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:36.763618 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:36.763498 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" event={"ID":"30a02ca9-8595-4ec2-a753-da9dceec7b49","Type":"ContainerStarted","Data":"9858750306b00fa8eea2f1c59e8e950ea72f494ca26f013d1e532d15afeab9f9"} Jul 10 18:03:37.287030 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.286997 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:37.287614 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.287576 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:37.302754 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.302705 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xpsbq" podStartSLOduration=4.195386577 podStartE2EDuration="25.302687407s" totalImagesPullingTime="21.10730083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:35.795444535 +0000 UTC m=+24.890635022" watchObservedRunningTime="2026-07-10 18:03:37.302687407 +0000 UTC m=+26.397877893" Jul 10 18:03:37.457885 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.457807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:37.458084 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.457806 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:37.458084 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:37.457944 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:37.458084 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:37.457995 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:37.769414 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.769334 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:03:37.769984 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.769814 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"f9c61fc48ebe204bd6db4e0e30c8d9b2a0b8dfa09e86d31e8ee8485b8d70316d"} Jul 10 18:03:37.772134 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.772100 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" event={"ID":"30a02ca9-8595-4ec2-a753-da9dceec7b49","Type":"ContainerStarted","Data":"7a94c9037da56ad3b3ad8a8ee3b6576d637f8e08761bc3b7443d33de8c4f2b6e"} Jul 10 18:03:37.772252 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.772236 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:37.772797 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.772776 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xp5fp" Jul 10 18:03:37.790279 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:37.790236 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvfds" podStartSLOduration=2.090014494 podStartE2EDuration="25.790218479s" totalImagesPullingTime="23.700203985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:37.789558812 +0000 UTC m=+26.884749311" watchObservedRunningTime="2026-07-10 18:03:37.790218479 +0000 UTC m=+26.885408967" Jul 10 18:03:38.457786 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:38.457753 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:38.457974 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:38.457908 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:39.458512 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:39.458284 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:39.459005 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:39.458577 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:39.459005 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:39.458354 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:39.459120 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:39.459078 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:39.475889 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:39.475860 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zt5vq_dbcb7ac8-3470-433d-9765-9cda8d1909f9/dns-node-resolver/0.log" Jul 10 18:03:40.251984 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.251878 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hjstd_e8c2b25e-ffaf-48b9-b544-fff1ab93f1da/node-ca/0.log" Jul 10 18:03:40.457649 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.457626 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:40.457750 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:40.457732 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:40.781783 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.781716 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:03:40.782556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.782059 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"d53ae9c75cbbfca78b4817c03d6f7acb51111ade815ea312dd49e85c28f63cbc"} Jul 10 18:03:40.782556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.782423 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:40.782556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.782490 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:40.782556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.782518 2570 scope.go:117] "RemoveContainer" containerID="f8d2625cb290debca2b0aa4913dab0322c2e00c07a1423c3e67da9b9c9b9b330" Jul 10 18:03:40.784059 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.784033 2570 generic.go:356] "Generic (PLEG): container finished" podID="32795b85-db8f-480d-9630-2d1d11a12bd3" containerID="4796d1d6aa25aa7f2db029bf5cd92fb02ebca661a78c6ec0a8a0ce1dca4d2df4" exitCode=0 Jul 10 18:03:40.784147 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.784084 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerDied","Data":"4796d1d6aa25aa7f2db029bf5cd92fb02ebca661a78c6ec0a8a0ce1dca4d2df4"} Jul 10 18:03:40.800487 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:40.800449 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:41.459766 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:41.459735 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:41.460045 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:41.459734 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:41.460045 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:41.459855 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:41.460045 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:41.459936 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:41.790342 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:41.790118 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:03:41.790724 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:41.790633 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" event={"ID":"974c1314-41cd-413c-adf4-0ca46ac529f4","Type":"ContainerStarted","Data":"bc5308225f5968ff0903ae96d7a1039978d0ee4c61972ad7329ea693d895bef6"} Jul 10 18:03:41.790836 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:41.790821 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:41.805936 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:41.805907 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:03:41.824318 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:41.824265 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" podStartSLOduration=8.636200572 podStartE2EDuration="29.824250826s" totalImagesPullingTime="21.188050254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:41.823406895 +0000 UTC m=+30.918597382" watchObservedRunningTime="2026-07-10 18:03:41.824250826 +0000 UTC m=+30.919441312" Jul 10 18:03:42.298709 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.298677 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cl6p6"] Jul 10 18:03:42.298872 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.298763 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:42.298872 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:42.298838 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:42.301939 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.301909 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kx74q"] Jul 10 18:03:42.302070 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.301999 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:42.302129 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:42.302092 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:42.302521 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.302505 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgjzs"] Jul 10 18:03:42.302590 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.302581 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:42.302689 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:42.302674 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:42.794810 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.794774 2570 generic.go:356] "Generic (PLEG): container finished" podID="32795b85-db8f-480d-9630-2d1d11a12bd3" containerID="40b7f80a5acf054232095917d5c0babb8e7f2082b1036548ee74178341c5ecf9" exitCode=0 Jul 10 18:03:42.795212 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:42.794900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerDied","Data":"40b7f80a5acf054232095917d5c0babb8e7f2082b1036548ee74178341c5ecf9"} Jul 10 18:03:43.458045 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:43.458011 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:43.458221 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:43.458017 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:43.458221 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:43.458143 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:43.458322 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:43.458243 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:44.071712 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:44.071626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:44.071712 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:44.071680 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:44.072124 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.071755 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:44.072124 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.071788 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:44.072124 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.071819 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs podName:c04d6277-39ba-474e-af1e-3f61c91741c6 nodeName:}" failed. No retries permitted until 2026-07-10 18:04:16.071801656 +0000 UTC m=+65.166992121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs") pod "network-metrics-daemon-mgjzs" (UID: "c04d6277-39ba-474e-af1e-3f61c91741c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:03:44.072124 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.071836 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret podName:ce68a2f5-08f1-47cf-80c8-004a8ac733bb nodeName:}" failed. No retries permitted until 2026-07-10 18:04:16.071828514 +0000 UTC m=+65.167018978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret") pod "global-pull-secret-syncer-cl6p6" (UID: "ce68a2f5-08f1-47cf-80c8-004a8ac733bb") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:03:44.172923 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:44.172898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:44.173055 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.173012 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:03:44.173055 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.173025 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:03:44.173055 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.173034 2570 projected.go:196] Error preparing data for projected volume kube-api-access-c9cjn for pod openshift-network-diagnostics/network-check-target-kx74q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:44.173174 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.173077 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn podName:7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2 nodeName:}" failed. No retries permitted until 2026-07-10 18:04:16.173065481 +0000 UTC m=+65.268255945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9cjn" (UniqueName: "kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn") pod "network-check-target-kx74q" (UID: "7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:03:44.457647 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:44.457617 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:44.457828 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:44.457724 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:44.802268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:44.802171 2570 generic.go:356] "Generic (PLEG): container finished" podID="32795b85-db8f-480d-9630-2d1d11a12bd3" containerID="9fa3522f0980aaf4ece4195470467507c7c46efcab69376e6c647e2a9a53b168" exitCode=0 Jul 10 18:03:44.802268 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:44.802244 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerDied","Data":"9fa3522f0980aaf4ece4195470467507c7c46efcab69376e6c647e2a9a53b168"} Jul 10 18:03:45.458418 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:45.458372 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:45.458881 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:45.458438 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:45.458881 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:45.458629 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:45.458881 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:45.458721 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:46.458026 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:46.457993 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:46.458175 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:46.458137 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:47.458064 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:47.458028 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:47.458505 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:47.458067 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:47.458505 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:47.458205 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:47.458505 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:47.458357 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:48.458530 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:48.458491 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:48.458981 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:48.458649 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:49.458439 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:49.458409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:49.458612 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:49.458420 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:49.458612 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:49.458524 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:49.458898 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:49.458633 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:49.817099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:49.817024 2570 generic.go:356] "Generic (PLEG): container finished" podID="32795b85-db8f-480d-9630-2d1d11a12bd3" containerID="1cce3a94535c2b87fef78369a36b726af2a23554fc2407ab336951613163aa6a" exitCode=0 Jul 10 18:03:49.817099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:49.817070 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerDied","Data":"1cce3a94535c2b87fef78369a36b726af2a23554fc2407ab336951613163aa6a"} Jul 10 18:03:50.457935 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:50.457901 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:50.458105 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:50.458000 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:50.822036 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:50.821962 2570 generic.go:356] "Generic (PLEG): container finished" podID="32795b85-db8f-480d-9630-2d1d11a12bd3" containerID="178735f28e3c29a35e348288215ff9d395b1ffb54fb69ca1e84f78bd93a66e03" exitCode=0 Jul 10 18:03:50.822036 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:50.822012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerDied","Data":"178735f28e3c29a35e348288215ff9d395b1ffb54fb69ca1e84f78bd93a66e03"} Jul 10 18:03:51.458350 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:51.458315 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:51.458505 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:51.458417 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:51.458620 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:51.458520 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:51.458685 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:51.458642 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:51.828174 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:51.828078 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wwslj" event={"ID":"32795b85-db8f-480d-9630-2d1d11a12bd3","Type":"ContainerStarted","Data":"9227c98ca4509ecf626efe76ca3ea148a26afea3d33c36f9806f127a616163ec"} Jul 10 18:03:51.850588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:51.850549 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wwslj" podStartSLOduration=6.478023606 podStartE2EDuration="39.850535894s" totalImagesPullingTime="33.372512288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:03:51.849285491 +0000 UTC m=+40.944475978" watchObservedRunningTime="2026-07-10 18:03:51.850535894 +0000 UTC m=+40.945726381" Jul 10 18:03:52.457712 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:52.457680 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:52.457875 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:52.457788 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:53.457609 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:53.457578 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:53.457977 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:53.457723 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:53.457977 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:53.457773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:53.457977 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:53.457855 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:54.457650 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:54.457623 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:54.457982 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:54.457724 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:55.458486 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:55.458445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:55.458887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:55.458445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:55.458887 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:55.458569 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:55.458887 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:55.458653 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:56.457837 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:56.457806 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:56.458010 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:56.457908 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:57.458242 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:57.458212 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:57.458700 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:57.458321 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:03:57.458700 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:57.458404 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:57.458700 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:57.458507 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:58.458085 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:58.458042 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:03:58.458350 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:58.458231 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:03:59.458226 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:59.458189 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:03:59.458435 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:59.458294 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:03:59.458435 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:03:59.458383 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:03:59.458852 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:03:59.458494 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:04:00.458534 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:00.458494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:04:00.458948 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:00.458623 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:04:01.458320 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:01.458286 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:04:01.458491 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:01.458392 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cl6p6" podUID="ce68a2f5-08f1-47cf-80c8-004a8ac733bb" Jul 10 18:04:01.458491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:01.458443 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:04:01.458641 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:01.458615 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx74q" podUID="7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2" Jul 10 18:04:02.457997 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.457959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:04:02.458116 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:02.458086 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjzs" podUID="c04d6277-39ba-474e-af1e-3f61c91741c6" Jul 10 18:04:02.679765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.679743 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-217.ec2.internal" event="NodeReady" Jul 10 18:04:02.680142 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.679874 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 18:04:02.728990 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.728957 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-685668c86c-fjk9j"] Jul 10 18:04:02.752162 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.752129 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.756321 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.756298 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 18:04:02.756469 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.756400 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 18:04:02.756675 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.756655 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 18:04:02.756763 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.756749 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4gf6\"" Jul 10 18:04:02.767575 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.767548 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 18:04:02.778910 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.778876 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685668c86c-fjk9j"] Jul 10 18:04:02.780143 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.780116 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pnzm2"] Jul 10 18:04:02.792577 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.792546 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-22fz5"] Jul 10 18:04:02.792718 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.792687 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.795704 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.795684 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 18:04:02.797821 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.797806 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 18:04:02.798258 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.798240 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kztg\"" Jul 10 18:04:02.802574 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.802556 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:02.805954 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.805937 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 18:04:02.806058 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.805988 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 18:04:02.806292 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.806276 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 18:04:02.806340 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.806276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cpkz8\"" Jul 10 18:04:02.810249 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.810234 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 18:04:02.818066 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818041 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqhg\" (UniqueName: \"kubernetes.io/projected/0a0cf7f6-971f-4aef-be0d-8511986c673a-kube-api-access-8sqhg\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.818138 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0cf7f6-971f-4aef-be0d-8511986c673a-tmp-dir\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.818138 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818118 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34705db0-c7e8-4758-be6a-9867202042d1-ca-trust-extracted\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.818138 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34705db0-c7e8-4758-be6a-9867202042d1-trusted-ca\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.818235 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34705db0-c7e8-4758-be6a-9867202042d1-image-registry-private-configuration\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.818235 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818165 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34705db0-c7e8-4758-be6a-9867202042d1-registry-certificates\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.818235 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818180 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34705db0-c7e8-4758-be6a-9867202042d1-installation-pull-secrets\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.818235 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6xk\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-kube-api-access-zh6xk\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.818351 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-registry-tls\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.818351 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a0cf7f6-971f-4aef-be0d-8511986c673a-metrics-tls\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.818351 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818345 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0cf7f6-971f-4aef-be0d-8511986c673a-config-volume\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.818438 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.818363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-bound-sa-token\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.823554 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.823532 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22fz5"] Jul 10 18:04:02.831799 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.831765 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnzm2"] Jul 10 18:04:02.850197 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.850162 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9drf9"] Jul 10 18:04:02.863892 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.863863 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:02.867273 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.867244 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 18:04:02.867439 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.867249 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-57cv4\"" Jul 10 18:04:02.867580 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.867341 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 18:04:02.867707 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.867382 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 18:04:02.868130 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.868112 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9drf9"] Jul 10 18:04:02.918865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.918824 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-registry-tls\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.918865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.918873 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a0cf7f6-971f-4aef-be0d-8511986c673a-metrics-tls\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.918894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/892cc365-9261-4418-bb36-bbdaa6bf1c4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.918921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfnn\" (UniqueName: \"kubernetes.io/projected/892cc365-9261-4418-bb36-bbdaa6bf1c4d-kube-api-access-2bfnn\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.918947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0cf7f6-971f-4aef-be0d-8511986c673a-config-volume\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.918971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-bound-sa-token\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.918995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/892cc365-9261-4418-bb36-bbdaa6bf1c4d-data-volume\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919010 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/892cc365-9261-4418-bb36-bbdaa6bf1c4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqhg\" (UniqueName: \"kubernetes.io/projected/0a0cf7f6-971f-4aef-be0d-8511986c673a-kube-api-access-8sqhg\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hz89\" (UniqueName: \"kubernetes.io/projected/381b690f-e076-4a9c-a3b9-a47422de7245-kube-api-access-9hz89\") pod \"ingress-canary-9drf9\" (UID: \"381b690f-e076-4a9c-a3b9-a47422de7245\") " pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:02.919146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0cf7f6-971f-4aef-be0d-8511986c673a-tmp-dir\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34705db0-c7e8-4758-be6a-9867202042d1-ca-trust-extracted\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34705db0-c7e8-4758-be6a-9867202042d1-trusted-ca\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34705db0-c7e8-4758-be6a-9867202042d1-image-registry-private-configuration\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34705db0-c7e8-4758-be6a-9867202042d1-registry-certificates\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919398 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34705db0-c7e8-4758-be6a-9867202042d1-installation-pull-secrets\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919419 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6xk\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-kube-api-access-zh6xk\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919441 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/892cc365-9261-4418-bb36-bbdaa6bf1c4d-crio-socket\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/381b690f-e076-4a9c-a3b9-a47422de7245-cert\") pod \"ingress-canary-9drf9\" (UID: \"381b690f-e076-4a9c-a3b9-a47422de7245\") " pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:02.919570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919571 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0cf7f6-971f-4aef-be0d-8511986c673a-tmp-dir\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.920028 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919707 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34705db0-c7e8-4758-be6a-9867202042d1-ca-trust-extracted\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.920028 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.919723 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0cf7f6-971f-4aef-be0d-8511986c673a-config-volume\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.920406 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.920364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34705db0-c7e8-4758-be6a-9867202042d1-registry-certificates\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.920564 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.920546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34705db0-c7e8-4758-be6a-9867202042d1-trusted-ca\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.923674 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.923651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a0cf7f6-971f-4aef-be0d-8511986c673a-metrics-tls\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.923768 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.923651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34705db0-c7e8-4758-be6a-9867202042d1-image-registry-private-configuration\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.923768 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.923653 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34705db0-c7e8-4758-be6a-9867202042d1-installation-pull-secrets\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.923768 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.923702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-registry-tls\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.927951 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.927925 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqhg\" (UniqueName: \"kubernetes.io/projected/0a0cf7f6-971f-4aef-be0d-8511986c673a-kube-api-access-8sqhg\") pod \"dns-default-pnzm2\" (UID: \"0a0cf7f6-971f-4aef-be0d-8511986c673a\") " pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:02.928433 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.928416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-bound-sa-token\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:02.928758 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:02.928730 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6xk\" (UniqueName: \"kubernetes.io/projected/34705db0-c7e8-4758-be6a-9867202042d1-kube-api-access-zh6xk\") pod \"image-registry-685668c86c-fjk9j\" (UID: \"34705db0-c7e8-4758-be6a-9867202042d1\") " pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:03.020767 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hz89\" (UniqueName: \"kubernetes.io/projected/381b690f-e076-4a9c-a3b9-a47422de7245-kube-api-access-9hz89\") pod \"ingress-canary-9drf9\" (UID: \"381b690f-e076-4a9c-a3b9-a47422de7245\") " pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:03.020767 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/892cc365-9261-4418-bb36-bbdaa6bf1c4d-crio-socket\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.020767 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/381b690f-e076-4a9c-a3b9-a47422de7245-cert\") pod \"ingress-canary-9drf9\" (UID: \"381b690f-e076-4a9c-a3b9-a47422de7245\") " pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:03.021042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/892cc365-9261-4418-bb36-bbdaa6bf1c4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.021042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfnn\" (UniqueName: \"kubernetes.io/projected/892cc365-9261-4418-bb36-bbdaa6bf1c4d-kube-api-access-2bfnn\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.021042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020829 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/892cc365-9261-4418-bb36-bbdaa6bf1c4d-data-volume\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.021042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020836 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/892cc365-9261-4418-bb36-bbdaa6bf1c4d-crio-socket\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.021042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.020891 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/892cc365-9261-4418-bb36-bbdaa6bf1c4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.021279 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.021189 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/892cc365-9261-4418-bb36-bbdaa6bf1c4d-data-volume\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.021351 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.021333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/892cc365-9261-4418-bb36-bbdaa6bf1c4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.023318 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.023298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/892cc365-9261-4418-bb36-bbdaa6bf1c4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.023558 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.023539 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/381b690f-e076-4a9c-a3b9-a47422de7245-cert\") pod \"ingress-canary-9drf9\" (UID: \"381b690f-e076-4a9c-a3b9-a47422de7245\") " pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:03.029294 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.029265 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfnn\" (UniqueName: \"kubernetes.io/projected/892cc365-9261-4418-bb36-bbdaa6bf1c4d-kube-api-access-2bfnn\") pod \"insights-runtime-extractor-22fz5\" (UID: \"892cc365-9261-4418-bb36-bbdaa6bf1c4d\") " pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.029389 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.029308 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hz89\" (UniqueName: \"kubernetes.io/projected/381b690f-e076-4a9c-a3b9-a47422de7245-kube-api-access-9hz89\") pod \"ingress-canary-9drf9\" (UID: \"381b690f-e076-4a9c-a3b9-a47422de7245\") " pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:03.061260 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.061218 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:03.101504 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.101477 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:03.120592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.120566 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22fz5" Jul 10 18:04:03.173825 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.173790 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9drf9" Jul 10 18:04:03.270763 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.270692 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685668c86c-fjk9j"] Jul 10 18:04:03.275207 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:03.275171 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34705db0_c7e8_4758_be6a_9867202042d1.slice/crio-3f22bd7d634566f175e0ca8df8a53b68fea6dca215c5dff1aabe41f721daff41 WatchSource:0}: Error finding container 3f22bd7d634566f175e0ca8df8a53b68fea6dca215c5dff1aabe41f721daff41: Status 404 returned error can't find the container with id 3f22bd7d634566f175e0ca8df8a53b68fea6dca215c5dff1aabe41f721daff41 Jul 10 18:04:03.291182 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.291151 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnzm2"] Jul 10 18:04:03.295064 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:03.295024 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0cf7f6_971f_4aef_be0d_8511986c673a.slice/crio-02a19903c8d9d3ae8443ca2c0bbb08ba9592d9ee3f8d25154af594b7c84eb67a WatchSource:0}: Error finding container 02a19903c8d9d3ae8443ca2c0bbb08ba9592d9ee3f8d25154af594b7c84eb67a: Status 404 returned error can't find the container with id 02a19903c8d9d3ae8443ca2c0bbb08ba9592d9ee3f8d25154af594b7c84eb67a Jul 10 18:04:03.306943 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.306436 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22fz5"] Jul 10 18:04:03.367492 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.365246 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9drf9"] Jul 10 18:04:03.370127 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:03.370100 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381b690f_e076_4a9c_a3b9_a47422de7245.slice/crio-f4cb4ef56c9ff3f3db1ada22b165efb7af5e58f204ace78dbaf51248472c6878 WatchSource:0}: Error finding container f4cb4ef56c9ff3f3db1ada22b165efb7af5e58f204ace78dbaf51248472c6878: Status 404 returned error can't find the container with id f4cb4ef56c9ff3f3db1ada22b165efb7af5e58f204ace78dbaf51248472c6878 Jul 10 18:04:03.457581 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.457546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:04:03.457772 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.457754 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:04:03.460367 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.460344 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:04:03.460563 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.460344 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7bsz9\"" Jul 10 18:04:03.460637 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.460586 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:04:03.460722 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.460704 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:04:03.858803 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.858723 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9drf9" event={"ID":"381b690f-e076-4a9c-a3b9-a47422de7245","Type":"ContainerStarted","Data":"f4cb4ef56c9ff3f3db1ada22b165efb7af5e58f204ace78dbaf51248472c6878"} Jul 10 18:04:03.860981 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.860908 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" event={"ID":"34705db0-c7e8-4758-be6a-9867202042d1","Type":"ContainerStarted","Data":"5e0aab3ed2c368d73e271c1e58a9f5a5bb77c38603a3561e7cfa9028e71835d2"} Jul 10 18:04:03.860981 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.860962 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" event={"ID":"34705db0-c7e8-4758-be6a-9867202042d1","Type":"ContainerStarted","Data":"3f22bd7d634566f175e0ca8df8a53b68fea6dca215c5dff1aabe41f721daff41"} Jul 10 18:04:03.861440 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.861250 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:03.862764 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.862736 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnzm2" event={"ID":"0a0cf7f6-971f-4aef-be0d-8511986c673a","Type":"ContainerStarted","Data":"02a19903c8d9d3ae8443ca2c0bbb08ba9592d9ee3f8d25154af594b7c84eb67a"} Jul 10 18:04:03.865578 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.865550 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22fz5" event={"ID":"892cc365-9261-4418-bb36-bbdaa6bf1c4d","Type":"ContainerStarted","Data":"96d1ee2f43888812984f1492537ab15948fe34b080e2cfaab10a1e9c0052c47a"} Jul 10 18:04:03.888355 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:03.888029 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" podStartSLOduration=1.8880084 podStartE2EDuration="1.8880084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:03.886762255 +0000 UTC m=+52.981952742" watchObservedRunningTime="2026-07-10 18:04:03.8880084 +0000 UTC m=+52.983198887" Jul 10 18:04:04.457908 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:04.457871 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:04:04.462341 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:04.461890 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb6vc\"" Jul 10 18:04:04.462341 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:04.462162 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:04:05.029650 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.028576 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c45bd8d8-kswn8"] Jul 10 18:04:05.048349 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.048289 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c45bd8d8-kswn8"] Jul 10 18:04:05.048510 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.048416 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.054100 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.054053 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 18:04:05.055707 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.055683 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 18:04:05.055980 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.055962 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 18:04:05.056258 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.056241 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hdtxr\"" Jul 10 18:04:05.056475 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.056460 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 18:04:05.056734 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.056719 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 18:04:05.056951 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.056936 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 18:04:05.057176 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.057160 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 18:04:05.141334 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.141237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-oauth-config\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.141334 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.141291 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-service-ca\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.141654 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.141395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-kube-api-access-5l5qb\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.141654 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.141496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-serving-cert\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.141654 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.141543 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-config\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.141654 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.141566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-oauth-serving-cert\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.242508 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.242473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-serving-cert\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.242714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.242534 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-config\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.242714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.242560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-oauth-serving-cert\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.242714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.242591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-oauth-config\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.242714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.242639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-service-ca\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.242714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.242694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-kube-api-access-5l5qb\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.243399 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.243373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-oauth-serving-cert\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.243506 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.243375 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-config\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.243919 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.243895 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-service-ca\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.246099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.246078 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-oauth-config\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.248389 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.248365 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-serving-cert\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.253045 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.253022 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-kube-api-access-5l5qb\") pod \"console-6c45bd8d8-kswn8\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:05.362233 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:05.362153 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:07.211276 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.211244 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6fccd774-5rk68"] Jul 10 18:04:07.294198 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.294167 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k"] Jul 10 18:04:07.294317 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.294292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.299565 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.299536 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k"] Jul 10 18:04:07.299771 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.299571 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6fccd774-5rk68"] Jul 10 18:04:07.299771 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.299677 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" Jul 10 18:04:07.302865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.302845 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sdzwp\"" Jul 10 18:04:07.302957 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.302945 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 18:04:07.303189 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.303174 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 18:04:07.362726 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-oauth-serving-cert\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.362870 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-config\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.362870 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-trusted-ca-bundle\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.362870 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362801 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-service-ca\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.362870 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-serving-cert\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.363062 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362872 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxnb\" (UniqueName: \"kubernetes.io/projected/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-kube-api-access-vhxnb\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.363062 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7af12151-b236-461f-a999-7615f8bfbdbe-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vk96k\" (UID: \"7af12151-b236-461f-a999-7615f8bfbdbe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" Jul 10 18:04:07.363062 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.362931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-oauth-config\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.463662 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-oauth-config\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.463816 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463693 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-oauth-serving-cert\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.463816 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-config\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.463816 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-trusted-ca-bundle\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.463816 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-service-ca\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.464021 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463827 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-serving-cert\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.464021 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxnb\" (UniqueName: \"kubernetes.io/projected/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-kube-api-access-vhxnb\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.464021 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.463898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7af12151-b236-461f-a999-7615f8bfbdbe-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vk96k\" (UID: \"7af12151-b236-461f-a999-7615f8bfbdbe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" Jul 10 18:04:07.464868 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.464641 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-oauth-serving-cert\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.464868 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.464650 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-config\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.464868 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.464715 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-service-ca\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.464868 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.464860 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-trusted-ca-bundle\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.466716 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.466694 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7af12151-b236-461f-a999-7615f8bfbdbe-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vk96k\" (UID: \"7af12151-b236-461f-a999-7615f8bfbdbe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" Jul 10 18:04:07.467232 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.467207 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-serving-cert\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.469862 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.469843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-oauth-config\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.474618 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.474578 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxnb\" (UniqueName: \"kubernetes.io/projected/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-kube-api-access-vhxnb\") pod \"console-5f6fccd774-5rk68\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.604726 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.604692 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:07.610718 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:07.610694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" Jul 10 18:04:09.130922 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.130567 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c45bd8d8-kswn8"] Jul 10 18:04:09.149282 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.149211 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k"] Jul 10 18:04:09.154273 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:09.154232 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af12151_b236_461f_a999_7615f8bfbdbe.slice/crio-818db11eef10e4a44d4895e36e23e55d6385df93451175c36f7113da6f7d2aca WatchSource:0}: Error finding container 818db11eef10e4a44d4895e36e23e55d6385df93451175c36f7113da6f7d2aca: Status 404 returned error can't find the container with id 818db11eef10e4a44d4895e36e23e55d6385df93451175c36f7113da6f7d2aca Jul 10 18:04:09.171837 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.171809 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6fccd774-5rk68"] Jul 10 18:04:09.176506 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:09.176478 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bd1832_fefd_4ea2_bd98_79ace7e16ea5.slice/crio-cc88e69d9033477657b6bd8bc829ae28ada8325a19bcc26f534a64bd2302617c WatchSource:0}: Error finding container cc88e69d9033477657b6bd8bc829ae28ada8325a19bcc26f534a64bd2302617c: Status 404 returned error can't find the container with id cc88e69d9033477657b6bd8bc829ae28ada8325a19bcc26f534a64bd2302617c Jul 10 18:04:09.886527 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.886013 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnzm2" event={"ID":"0a0cf7f6-971f-4aef-be0d-8511986c673a","Type":"ContainerStarted","Data":"0deeffc6e251e3c5e8a1945139e1e8602f464fc4f462eadbba2fe60119a3695a"} Jul 10 18:04:09.886527 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.886080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnzm2" event={"ID":"0a0cf7f6-971f-4aef-be0d-8511986c673a","Type":"ContainerStarted","Data":"4a3247577a8824cfc5ed78d109e886366e0c4a9f1f2081d3ea26e67080d5bb8e"} Jul 10 18:04:09.886527 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.886157 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:09.887577 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.887506 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45bd8d8-kswn8" event={"ID":"f2cbe0d2-c025-4b67-9c2c-36122525f8d0","Type":"ContainerStarted","Data":"9353a8aa69a6597b725e27e97e8348586012588041d03e7ca659bbd7f9286a1f"} Jul 10 18:04:09.890408 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.890378 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22fz5" event={"ID":"892cc365-9261-4418-bb36-bbdaa6bf1c4d","Type":"ContainerStarted","Data":"d0457131eaaf6542e3d09e098b12c102cb7f44c88677fc301c3dd6a788be7353"} Jul 10 18:04:09.890565 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.890432 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22fz5" event={"ID":"892cc365-9261-4418-bb36-bbdaa6bf1c4d","Type":"ContainerStarted","Data":"15b616a321ba64038622ad61c984725b9166a09c1b4ce735132bf158842d7cab"} Jul 10 18:04:09.892856 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.892825 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9drf9" event={"ID":"381b690f-e076-4a9c-a3b9-a47422de7245","Type":"ContainerStarted","Data":"b687ea7a6c2417838956ce16b97879c0d8dd00f21099f6b1e929152f582e7b3a"} Jul 10 18:04:09.894573 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.894544 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" event={"ID":"7af12151-b236-461f-a999-7615f8bfbdbe","Type":"ContainerStarted","Data":"818db11eef10e4a44d4895e36e23e55d6385df93451175c36f7113da6f7d2aca"} Jul 10 18:04:09.895879 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.895857 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fccd774-5rk68" event={"ID":"53bd1832-fefd-4ea2-bd98-79ace7e16ea5","Type":"ContainerStarted","Data":"cc88e69d9033477657b6bd8bc829ae28ada8325a19bcc26f534a64bd2302617c"} Jul 10 18:04:09.910029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.909286 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pnzm2" podStartSLOduration=2.2518828920000002 podStartE2EDuration="7.909265669s" totalImagesPullingTime="5.657382777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:09.908695715 +0000 UTC m=+59.003886202" watchObservedRunningTime="2026-07-10 18:04:09.909265669 +0000 UTC m=+59.004456156" Jul 10 18:04:09.937118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:09.936996 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9drf9" podStartSLOduration=2.332556083 podStartE2EDuration="7.936971323s" totalImagesPullingTime="5.60441524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:09.936484638 +0000 UTC m=+59.031675124" watchObservedRunningTime="2026-07-10 18:04:09.936971323 +0000 UTC m=+59.032161813" Jul 10 18:04:10.903101 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:10.902999 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" event={"ID":"7af12151-b236-461f-a999-7615f8bfbdbe","Type":"ContainerStarted","Data":"08d454693ef623730fbbe7227b8e9fa848dd5a2ced1a3fa8d42489e70df3c62c"} Jul 10 18:04:10.921330 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:10.920584 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" podStartSLOduration=2.653790315 podStartE2EDuration="3.920561652s" totalImagesPullingTime="1.266771337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:10.919933014 +0000 UTC m=+60.015123500" watchObservedRunningTime="2026-07-10 18:04:10.920561652 +0000 UTC m=+60.015752140" Jul 10 18:04:11.906566 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:11.906527 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" Jul 10 18:04:11.912300 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:11.912271 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vk96k" Jul 10 18:04:13.816964 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:13.816930 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6675h" Jul 10 18:04:14.918793 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:14.918755 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fccd774-5rk68" event={"ID":"53bd1832-fefd-4ea2-bd98-79ace7e16ea5","Type":"ContainerStarted","Data":"2ce3082b016e574f7b1ab8691022468fc7823992a2004672e78adcfcd58aefe8"} Jul 10 18:04:14.920366 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:14.920343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45bd8d8-kswn8" event={"ID":"f2cbe0d2-c025-4b67-9c2c-36122525f8d0","Type":"ContainerStarted","Data":"457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5"} Jul 10 18:04:14.922205 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:14.922155 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22fz5" event={"ID":"892cc365-9261-4418-bb36-bbdaa6bf1c4d","Type":"ContainerStarted","Data":"a58214e0bf6e53df5ce894bcddeeefc628c66865090f9c9a55d9a6937822be97"} Jul 10 18:04:14.938778 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:14.938733 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f6fccd774-5rk68" podStartSLOduration=2.928478307 podStartE2EDuration="7.938719814s" totalImagesPullingTime="5.010241507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:14.937539014 +0000 UTC m=+64.032729499" watchObservedRunningTime="2026-07-10 18:04:14.938719814 +0000 UTC m=+64.033910371" Jul 10 18:04:14.956538 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:14.956495 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-22fz5" podStartSLOduration=2.265227535 podStartE2EDuration="12.956481443s" totalImagesPullingTime="10.691253908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:04:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:14.955344374 +0000 UTC m=+64.050534859" watchObservedRunningTime="2026-07-10 18:04:14.956481443 +0000 UTC m=+64.051671970" Jul 10 18:04:14.979129 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:14.979089 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c45bd8d8-kswn8" podStartSLOduration=5.929872634 podStartE2EDuration="10.979075646s" totalImagesPullingTime="5.049203012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:14.977585753 +0000 UTC m=+64.072776238" watchObservedRunningTime="2026-07-10 18:04:14.979075646 +0000 UTC m=+64.074266131" Jul 10 18:04:15.362421 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:15.362285 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:15.362421 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:15.362386 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:15.363692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:15.363665 2570 patch_prober.go:28] interesting pod/console-6c45bd8d8-kswn8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body= Jul 10 18:04:15.363745 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:15.363719 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6c45bd8d8-kswn8" podUID="f2cbe0d2-c025-4b67-9c2c-36122525f8d0" containerName="console" probeResult="failure" output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" Jul 10 18:04:16.136101 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.136050 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:04:16.136485 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.136127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:04:16.139228 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.139210 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:04:16.140138 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.140124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:04:16.148571 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.148545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04d6277-39ba-474e-af1e-3f61c91741c6-metrics-certs\") pod \"network-metrics-daemon-mgjzs\" (UID: \"c04d6277-39ba-474e-af1e-3f61c91741c6\") " pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:04:16.148571 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.148558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce68a2f5-08f1-47cf-80c8-004a8ac733bb-original-pull-secret\") pod \"global-pull-secret-syncer-cl6p6\" (UID: \"ce68a2f5-08f1-47cf-80c8-004a8ac733bb\") " pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:04:16.172949 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.172923 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb6vc\"" Jul 10 18:04:16.180981 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.180964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjzs" Jul 10 18:04:16.236790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.236755 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:04:16.240112 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.240040 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:04:16.249888 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.249865 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:04:16.260818 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.260729 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cjn\" (UniqueName: \"kubernetes.io/projected/7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2-kube-api-access-c9cjn\") pod \"network-check-target-kx74q\" (UID: \"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2\") " pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:04:16.307087 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.307044 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgjzs"] Jul 10 18:04:16.310001 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:16.309973 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04d6277_39ba_474e_af1e_3f61c91741c6.slice/crio-65dba800ec5b29b9ea62369e06e00aab5625eb3bf58cd5aa00291e671b982e2f WatchSource:0}: Error finding container 65dba800ec5b29b9ea62369e06e00aab5625eb3bf58cd5aa00291e671b982e2f: Status 404 returned error can't find the container with id 65dba800ec5b29b9ea62369e06e00aab5625eb3bf58cd5aa00291e671b982e2f Jul 10 18:04:16.367620 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.367553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cl6p6" Jul 10 18:04:16.375074 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.375043 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7bsz9\"" Jul 10 18:04:16.383043 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.383019 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:04:16.504869 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.504832 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cl6p6"] Jul 10 18:04:16.508317 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:16.508283 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce68a2f5_08f1_47cf_80c8_004a8ac733bb.slice/crio-943083d7296ddaa711e9cddca71cf54e517bf41168e268b360205efb20bcab74 WatchSource:0}: Error finding container 943083d7296ddaa711e9cddca71cf54e517bf41168e268b360205efb20bcab74: Status 404 returned error can't find the container with id 943083d7296ddaa711e9cddca71cf54e517bf41168e268b360205efb20bcab74 Jul 10 18:04:16.522740 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.522710 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kx74q"] Jul 10 18:04:16.525904 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:16.525878 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9dac56_bede_4fee_bd7f_ddedf4e2b3f2.slice/crio-5fb2e80afec6ae9b0bf43b0ace40a787f092ddbe664fbc73fd00c80736f8d5d9 WatchSource:0}: Error finding container 5fb2e80afec6ae9b0bf43b0ace40a787f092ddbe664fbc73fd00c80736f8d5d9: Status 404 returned error can't find the container with id 5fb2e80afec6ae9b0bf43b0ace40a787f092ddbe664fbc73fd00c80736f8d5d9 Jul 10 18:04:16.936557 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.936512 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgjzs" event={"ID":"c04d6277-39ba-474e-af1e-3f61c91741c6","Type":"ContainerStarted","Data":"65dba800ec5b29b9ea62369e06e00aab5625eb3bf58cd5aa00291e671b982e2f"} Jul 10 18:04:16.938234 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.938204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kx74q" event={"ID":"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2","Type":"ContainerStarted","Data":"5fb2e80afec6ae9b0bf43b0ace40a787f092ddbe664fbc73fd00c80736f8d5d9"} Jul 10 18:04:16.939960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:16.939930 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cl6p6" event={"ID":"ce68a2f5-08f1-47cf-80c8-004a8ac733bb","Type":"ContainerStarted","Data":"943083d7296ddaa711e9cddca71cf54e517bf41168e268b360205efb20bcab74"} Jul 10 18:04:17.605729 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:17.605073 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:17.606826 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:17.606772 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:17.608524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:17.608104 2570 patch_prober.go:28] interesting pod/console-5f6fccd774-5rk68 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body= Jul 10 18:04:17.608524 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:17.608155 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5f6fccd774-5rk68" podUID="53bd1832-fefd-4ea2-bd98-79ace7e16ea5" containerName="console" probeResult="failure" output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" Jul 10 18:04:17.947137 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:17.947094 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgjzs" event={"ID":"c04d6277-39ba-474e-af1e-3f61c91741c6","Type":"ContainerStarted","Data":"e4317d49914e124dcbed3c12d7e86d4ecbc39f6dea86cec80bca8b96f88a7bd8"} Jul 10 18:04:17.947323 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:17.947163 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgjzs" event={"ID":"c04d6277-39ba-474e-af1e-3f61c91741c6","Type":"ContainerStarted","Data":"b57db59add04c2cbdc893be766178a2489460ee555a3322c15bae6f941c83fde"} Jul 10 18:04:17.964322 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:17.964264 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mgjzs" podStartSLOduration=64.860873237 podStartE2EDuration="1m5.964249039s" totalImagesPullingTime="1.103375802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:17.963034893 +0000 UTC m=+67.058225376" watchObservedRunningTime="2026-07-10 18:04:17.964249039 +0000 UTC m=+67.059439526" Jul 10 18:04:19.714464 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.714432 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8686k"] Jul 10 18:04:19.717109 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.717077 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.719823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.719802 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 18:04:19.719945 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.719930 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 18:04:19.720089 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.720072 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 18:04:19.720171 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.720154 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 18:04:19.721219 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.721072 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 18:04:19.721219 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.721116 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vwlhd\"" Jul 10 18:04:19.721219 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.721116 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 18:04:19.764807 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.764783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-textfile\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.764941 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.764811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.764941 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.764833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-root\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.764941 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.764858 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jm4\" (UniqueName: \"kubernetes.io/projected/14c66d29-b215-466b-a6b7-cd0451b307bb-kube-api-access-g9jm4\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.765066 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.764941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-sys\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.765066 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.764970 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.765066 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.764990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.765066 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.765016 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-wtmp\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.765066 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.765040 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c66d29-b215-466b-a6b7-cd0451b307bb-metrics-client-ca\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866413 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c66d29-b215-466b-a6b7-cd0451b307bb-metrics-client-ca\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866413 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-textfile\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866413 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866403 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-root\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866462 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jm4\" (UniqueName: \"kubernetes.io/projected/14c66d29-b215-466b-a6b7-cd0451b307bb-kube-api-access-g9jm4\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866493 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-sys\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866696 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:19.866533 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 18:04:19.866696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866547 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866696 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:19.866636 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls podName:14c66d29-b215-466b-a6b7-cd0451b307bb nodeName:}" failed. No retries permitted until 2026-07-10 18:04:20.366584059 +0000 UTC m=+69.461774552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls") pod "node-exporter-8686k" (UID: "14c66d29-b215-466b-a6b7-cd0451b307bb") : secret "node-exporter-tls" not found Jul 10 18:04:19.866994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866725 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-wtmp\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866775 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-textfile\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-root\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866858 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-sys\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.866994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.866893 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-wtmp\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.867178 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.867077 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c66d29-b215-466b-a6b7-cd0451b307bb-metrics-client-ca\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.867351 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.867332 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.869656 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.869634 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.877438 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.877415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jm4\" (UniqueName: \"kubernetes.io/projected/14c66d29-b215-466b-a6b7-cd0451b307bb-kube-api-access-g9jm4\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:19.906331 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.906305 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pnzm2" Jul 10 18:04:19.961794 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.961749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kx74q" event={"ID":"7c9dac56-bede-4fee-bd7f-ddedf4e2b3f2","Type":"ContainerStarted","Data":"34f5f797fb667c7337ad65486720eb8cd50a9d450756182d7b205a66820f91d4"} Jul 10 18:04:19.962130 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.962108 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:04:19.981346 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:19.981287 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kx74q" podStartSLOduration=64.969490643 podStartE2EDuration="1m7.981267336s" totalImagesPullingTime="3.011776693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:19.980627346 +0000 UTC m=+69.075817833" watchObservedRunningTime="2026-07-10 18:04:19.981267336 +0000 UTC m=+69.076457842" Jul 10 18:04:20.371797 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:20.371757 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:20.371996 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:20.371932 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 18:04:20.372058 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:04:20.372011 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls podName:14c66d29-b215-466b-a6b7-cd0451b307bb nodeName:}" failed. No retries permitted until 2026-07-10 18:04:21.371991622 +0000 UTC m=+70.467182093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls") pod "node-exporter-8686k" (UID: "14c66d29-b215-466b-a6b7-cd0451b307bb") : secret "node-exporter-tls" not found Jul 10 18:04:21.386109 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:21.386052 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:21.388571 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:21.388546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c66d29-b215-466b-a6b7-cd0451b307bb-node-exporter-tls\") pod \"node-exporter-8686k\" (UID: \"14c66d29-b215-466b-a6b7-cd0451b307bb\") " pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:21.528551 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:21.528522 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8686k" Jul 10 18:04:21.693066 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:21.693039 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c66d29_b215_466b_a6b7_cd0451b307bb.slice/crio-5b4eea424e4b0e03e5ebb5cba74af4b2d85b69b8fda3560deada0fbbfbe9d0c7 WatchSource:0}: Error finding container 5b4eea424e4b0e03e5ebb5cba74af4b2d85b69b8fda3560deada0fbbfbe9d0c7: Status 404 returned error can't find the container with id 5b4eea424e4b0e03e5ebb5cba74af4b2d85b69b8fda3560deada0fbbfbe9d0c7 Jul 10 18:04:21.970618 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:21.970502 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cl6p6" event={"ID":"ce68a2f5-08f1-47cf-80c8-004a8ac733bb","Type":"ContainerStarted","Data":"c6368808ee8be99656c929f2ae50953d3853a7d85a8a89c30c51c8a55644bffb"} Jul 10 18:04:21.971682 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:21.971657 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8686k" event={"ID":"14c66d29-b215-466b-a6b7-cd0451b307bb","Type":"ContainerStarted","Data":"5b4eea424e4b0e03e5ebb5cba74af4b2d85b69b8fda3560deada0fbbfbe9d0c7"} Jul 10 18:04:21.986916 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:21.986872 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cl6p6" podStartSLOduration=64.797664029 podStartE2EDuration="1m9.986859169s" totalImagesPullingTime="5.18919514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:21.98537584 +0000 UTC m=+71.080566326" watchObservedRunningTime="2026-07-10 18:04:21.986859169 +0000 UTC m=+71.082049654" Jul 10 18:04:22.977251 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:22.977213 2570 generic.go:356] "Generic (PLEG): container finished" podID="14c66d29-b215-466b-a6b7-cd0451b307bb" containerID="7b62f8ca847265938dd546b1ef447f0b484a9fa7d94adfd3392622a0e598b849" exitCode=0 Jul 10 18:04:22.977586 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:22.977307 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8686k" event={"ID":"14c66d29-b215-466b-a6b7-cd0451b307bb","Type":"ContainerDied","Data":"7b62f8ca847265938dd546b1ef447f0b484a9fa7d94adfd3392622a0e598b849"} Jul 10 18:04:23.066149 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.066124 2570 patch_prober.go:28] interesting pod/image-registry-685668c86c-fjk9j container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 18:04:23.066254 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.066168 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" podUID="34705db0-c7e8-4758-be6a-9867202042d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 18:04:23.965892 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.965852 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-689b6dc5d-rnw2l"] Jul 10 18:04:23.967941 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.967918 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:23.971007 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.970973 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 18:04:23.972172 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.972153 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 18:04:23.972292 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.972174 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 18:04:23.972292 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.972177 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4nbdkmifoun3j\"" Jul 10 18:04:23.972292 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.972153 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 18:04:23.972448 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.972314 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-drpj8\"" Jul 10 18:04:23.982717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.982688 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8686k" event={"ID":"14c66d29-b215-466b-a6b7-cd0451b307bb","Type":"ContainerStarted","Data":"84b5e28c8253a98f30e3ee17892e9df3951c57dd06c5ee93bbd289c2bc7e3560"} Jul 10 18:04:23.982991 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.982733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8686k" event={"ID":"14c66d29-b215-466b-a6b7-cd0451b307bb","Type":"ContainerStarted","Data":"1fc66cbc0fff3e3bd3d91ecc83e8359346d187fd8524a9cf70a9e9476afe71a4"} Jul 10 18:04:23.984761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:23.984743 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-689b6dc5d-rnw2l"] Jul 10 18:04:24.025728 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.025657 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8686k" podStartSLOduration=3.918208656 podStartE2EDuration="5.025638895s" totalImagesPullingTime="1.035280064s" totalInitContainerRuntime="72.150175ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:24.023965882 +0000 UTC m=+73.119156369" watchObservedRunningTime="2026-07-10 18:04:24.025638895 +0000 UTC m=+73.120829381" Jul 10 18:04:24.106942 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.106901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3df3abe4-98ff-4030-ba73-7a3938448563-audit-log\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.107173 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.106955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5zg\" (UniqueName: \"kubernetes.io/projected/3df3abe4-98ff-4030-ba73-7a3938448563-kube-api-access-7r5zg\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.107173 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.107030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-secret-metrics-server-tls\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.107173 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.107159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-client-ca-bundle\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.107298 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.107203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-secret-metrics-server-client-certs\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.107298 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.107229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3df3abe4-98ff-4030-ba73-7a3938448563-metrics-server-audit-profiles\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.107298 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.107282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3df3abe4-98ff-4030-ba73-7a3938448563-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208266 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-client-ca-bundle\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208266 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208275 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-secret-metrics-server-client-certs\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208529 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3df3abe4-98ff-4030-ba73-7a3938448563-metrics-server-audit-profiles\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208529 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3df3abe4-98ff-4030-ba73-7a3938448563-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208529 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3df3abe4-98ff-4030-ba73-7a3938448563-audit-log\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208529 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5zg\" (UniqueName: \"kubernetes.io/projected/3df3abe4-98ff-4030-ba73-7a3938448563-kube-api-access-7r5zg\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208529 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-secret-metrics-server-tls\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.208934 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.208896 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3df3abe4-98ff-4030-ba73-7a3938448563-audit-log\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.209421 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.209400 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3df3abe4-98ff-4030-ba73-7a3938448563-metrics-server-audit-profiles\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.209504 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.209456 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3df3abe4-98ff-4030-ba73-7a3938448563-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.211169 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.211147 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-secret-metrics-server-client-certs\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.211247 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.211173 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-secret-metrics-server-tls\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.211285 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.211244 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3df3abe4-98ff-4030-ba73-7a3938448563-client-ca-bundle\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.216906 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.216846 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5zg\" (UniqueName: \"kubernetes.io/projected/3df3abe4-98ff-4030-ba73-7a3938448563-kube-api-access-7r5zg\") pod \"metrics-server-689b6dc5d-rnw2l\" (UID: \"3df3abe4-98ff-4030-ba73-7a3938448563\") " pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.277673 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.277638 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:24.406852 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.406703 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-689b6dc5d-rnw2l"] Jul 10 18:04:24.409266 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:24.409237 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df3abe4_98ff_4030_ba73_7a3938448563.slice/crio-9d11b8a0c36bcb5351161fa8d258103716fe358f335e6f8c18d4d0d7dd31c214 WatchSource:0}: Error finding container 9d11b8a0c36bcb5351161fa8d258103716fe358f335e6f8c18d4d0d7dd31c214: Status 404 returned error can't find the container with id 9d11b8a0c36bcb5351161fa8d258103716fe358f335e6f8c18d4d0d7dd31c214 Jul 10 18:04:24.874354 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.874329 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-685668c86c-fjk9j" Jul 10 18:04:24.880142 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.880115 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr"] Jul 10 18:04:24.883492 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.883472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:24.885990 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.885969 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 18:04:24.886207 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.886192 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 18:04:24.886406 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.886383 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 18:04:24.886520 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.886417 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-v2qgx\"" Jul 10 18:04:24.886520 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.886492 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 18:04:24.886520 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.886505 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 18:04:24.892053 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.892029 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 18:04:24.898812 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.898790 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr"] Jul 10 18:04:24.987568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:24.987532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" event={"ID":"3df3abe4-98ff-4030-ba73-7a3938448563","Type":"ContainerStarted","Data":"9d11b8a0c36bcb5351161fa8d258103716fe358f335e6f8c18d4d0d7dd31c214"} Jul 10 18:04:25.015918 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.015884 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-serving-certs-ca-bundle\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.015918 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.015917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-federate-client-tls\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.016157 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.015987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-telemeter-client-tls\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.016157 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.016050 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbkt\" (UniqueName: \"kubernetes.io/projected/511ef8f2-df2d-473a-87b7-d61081033910-kube-api-access-bnbkt\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.016157 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.016082 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-metrics-client-ca\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.016157 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.016109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.016320 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.016218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.016320 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.016248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-secret-telemeter-client\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.117779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.117246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-serving-certs-ca-bundle\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.117779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.117292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-federate-client-tls\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.117779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.117470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-telemeter-client-tls\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.117779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.117556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbkt\" (UniqueName: \"kubernetes.io/projected/511ef8f2-df2d-473a-87b7-d61081033910-kube-api-access-bnbkt\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.117779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.117626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-metrics-client-ca\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.117779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.117653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.119023 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.118993 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.119574 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.119548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.119720 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.119592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-secret-telemeter-client\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.119933 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.119915 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-serving-certs-ca-bundle\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.120104 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.120079 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511ef8f2-df2d-473a-87b7-d61081033910-metrics-client-ca\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.121322 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.121302 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-federate-client-tls\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.123114 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.123080 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.125883 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.125814 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-telemeter-client-tls\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.126081 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.126063 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/511ef8f2-df2d-473a-87b7-d61081033910-secret-telemeter-client\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.130785 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.130763 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbkt\" (UniqueName: \"kubernetes.io/projected/511ef8f2-df2d-473a-87b7-d61081033910-kube-api-access-bnbkt\") pod \"telemeter-client-6dbfc976f4-q8wzr\" (UID: \"511ef8f2-df2d-473a-87b7-d61081033910\") " pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.193581 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.193543 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" Jul 10 18:04:25.358562 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.358429 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr"] Jul 10 18:04:25.362249 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:25.362219 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511ef8f2_df2d_473a_87b7_d61081033910.slice/crio-5042fbe61ad11df9d25e9e3a7544b787d0a173688fe8f0e2398bc07bb8df9e01 WatchSource:0}: Error finding container 5042fbe61ad11df9d25e9e3a7544b787d0a173688fe8f0e2398bc07bb8df9e01: Status 404 returned error can't find the container with id 5042fbe61ad11df9d25e9e3a7544b787d0a173688fe8f0e2398bc07bb8df9e01 Jul 10 18:04:25.367537 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.367512 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:25.372824 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.372793 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:04:25.599073 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.599032 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6fccd774-5rk68"] Jul 10 18:04:25.993431 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:25.993384 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" event={"ID":"511ef8f2-df2d-473a-87b7-d61081033910","Type":"ContainerStarted","Data":"5042fbe61ad11df9d25e9e3a7544b787d0a173688fe8f0e2398bc07bb8df9e01"} Jul 10 18:04:26.999244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:26.999204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" event={"ID":"3df3abe4-98ff-4030-ba73-7a3938448563","Type":"ContainerStarted","Data":"48dae76ef23b372f5bddd4fafec722ac3853ee67e844af1866f2a110698f8fc9"} Jul 10 18:04:27.018504 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:27.018456 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" podStartSLOduration=2.357788371 podStartE2EDuration="4.018441532s" totalImagesPullingTime="1.660653161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:27.017434799 +0000 UTC m=+76.112625292" watchObservedRunningTime="2026-07-10 18:04:27.018441532 +0000 UTC m=+76.113632017" Jul 10 18:04:28.004466 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:28.004435 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" event={"ID":"511ef8f2-df2d-473a-87b7-d61081033910","Type":"ContainerStarted","Data":"b33e4b0aba2c77fb68d554cacfb20c360d0e939e3e993a534c9cc66ae6d6888e"} Jul 10 18:04:29.010506 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.010474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" event={"ID":"511ef8f2-df2d-473a-87b7-d61081033910","Type":"ContainerStarted","Data":"a56b51854e14dade18e8f6a98d36ebf84d53d121f49e9e0698ba0717b5a4e47f"} Jul 10 18:04:29.010906 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.010529 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" event={"ID":"511ef8f2-df2d-473a-87b7-d61081033910","Type":"ContainerStarted","Data":"1f502e4af58c97a7dedc6c13a7862a8a6a8f901c8187b81e794677af134e3831"} Jul 10 18:04:29.034736 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.034686 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dbfc976f4-q8wzr" podStartSLOduration=1.8953224340000001 podStartE2EDuration="5.034670548s" totalImagesPullingTime="3.139348114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:29.032571319 +0000 UTC m=+78.127761818" watchObservedRunningTime="2026-07-10 18:04:29.034670548 +0000 UTC m=+78.129861033" Jul 10 18:04:29.851654 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.851617 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65d958895d-5ndmm"] Jul 10 18:04:29.853686 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.853671 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:29.871905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.871875 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d958895d-5ndmm"] Jul 10 18:04:29.959738 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.959703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-config\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:29.959888 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.959742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-service-ca\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:29.959888 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.959770 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-oauth-config\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:29.959888 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.959852 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-trusted-ca-bundle\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:29.959988 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.959890 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-oauth-serving-cert\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:29.959988 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.959915 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtjz\" (UniqueName: \"kubernetes.io/projected/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-kube-api-access-pmtjz\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:29.960054 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:29.959990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-serving-cert\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.061300 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.061274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-serving-cert\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.061732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.061310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-config\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.061732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.061336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-service-ca\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.061732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.061365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-oauth-config\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.061732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.061403 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-trusted-ca-bundle\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.061732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.061437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-oauth-serving-cert\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.061732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.061467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtjz\" (UniqueName: \"kubernetes.io/projected/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-kube-api-access-pmtjz\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.062183 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.062154 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-service-ca\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.062270 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.062161 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-config\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.062431 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.062413 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-trusted-ca-bundle\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.062488 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.062463 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-oauth-serving-cert\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.063924 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.063907 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-serving-cert\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.064030 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.064012 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-oauth-config\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.069364 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.069343 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtjz\" (UniqueName: \"kubernetes.io/projected/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-kube-api-access-pmtjz\") pod \"console-65d958895d-5ndmm\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.162404 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.162372 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:30.293080 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:30.293050 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d958895d-5ndmm"] Jul 10 18:04:30.296186 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:04:30.296162 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ec763d_a5c9_438c_9b2e_4ef63a55ba4f.slice/crio-4ac1bf282d5ea750f5ed5d7de3fca52f06f31f2cd0fdd5132454080cd447e53f WatchSource:0}: Error finding container 4ac1bf282d5ea750f5ed5d7de3fca52f06f31f2cd0fdd5132454080cd447e53f: Status 404 returned error can't find the container with id 4ac1bf282d5ea750f5ed5d7de3fca52f06f31f2cd0fdd5132454080cd447e53f Jul 10 18:04:31.019875 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:31.019833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d958895d-5ndmm" event={"ID":"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f","Type":"ContainerStarted","Data":"e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6"} Jul 10 18:04:31.020075 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:31.019902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d958895d-5ndmm" event={"ID":"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f","Type":"ContainerStarted","Data":"4ac1bf282d5ea750f5ed5d7de3fca52f06f31f2cd0fdd5132454080cd447e53f"} Jul 10 18:04:31.043178 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:31.043116 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65d958895d-5ndmm" podStartSLOduration=2.043098286 podStartE2EDuration="2.043098286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:04:31.042973251 +0000 UTC m=+80.138163737" watchObservedRunningTime="2026-07-10 18:04:31.043098286 +0000 UTC m=+80.138288772" Jul 10 18:04:40.163580 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:40.163465 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:40.163580 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:40.163539 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:40.168087 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:40.168061 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:41.057405 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:41.057378 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:04:41.185139 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:41.185109 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c45bd8d8-kswn8"] Jul 10 18:04:43.764189 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:43.764146 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9drf9_381b690f-e076-4a9c-a3b9-a47422de7245/serve-healthcheck-canary/0.log" Jul 10 18:04:44.277923 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:44.277882 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:44.278079 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:44.277951 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:04:50.622547 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:50.622504 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f6fccd774-5rk68" podUID="53bd1832-fefd-4ea2-bd98-79ace7e16ea5" containerName="console" containerID="cri-o://2ce3082b016e574f7b1ab8691022468fc7823992a2004672e78adcfcd58aefe8" gracePeriod=15 Jul 10 18:04:50.968681 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:50.968652 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kx74q" Jul 10 18:04:51.088339 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.088303 2570 generic.go:356] "Generic (PLEG): container finished" podID="53bd1832-fefd-4ea2-bd98-79ace7e16ea5" containerID="2ce3082b016e574f7b1ab8691022468fc7823992a2004672e78adcfcd58aefe8" exitCode=0 Jul 10 18:04:51.088496 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.088362 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fccd774-5rk68" event={"ID":"53bd1832-fefd-4ea2-bd98-79ace7e16ea5","Type":"ContainerDied","Data":"2ce3082b016e574f7b1ab8691022468fc7823992a2004672e78adcfcd58aefe8"} Jul 10 18:04:51.386523 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.386491 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:51.416254 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416226 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-service-ca\") pod \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " Jul 10 18:04:51.416407 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416285 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-oauth-serving-cert\") pod \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " Jul 10 18:04:51.416407 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416338 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhxnb\" (UniqueName: \"kubernetes.io/projected/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-kube-api-access-vhxnb\") pod \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " Jul 10 18:04:51.416407 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416374 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-serving-cert\") pod \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " Jul 10 18:04:51.416584 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416441 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-oauth-config\") pod \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " Jul 10 18:04:51.416584 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416513 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-config\") pod \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " Jul 10 18:04:51.416584 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416538 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-trusted-ca-bundle\") pod \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\" (UID: \"53bd1832-fefd-4ea2-bd98-79ace7e16ea5\") " Jul 10 18:04:51.416763 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416697 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-service-ca" (OuterVolumeSpecName: "service-ca") pod "53bd1832-fefd-4ea2-bd98-79ace7e16ea5" (UID: "53bd1832-fefd-4ea2-bd98-79ace7e16ea5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:04:51.416815 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416758 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "53bd1832-fefd-4ea2-bd98-79ace7e16ea5" (UID: "53bd1832-fefd-4ea2-bd98-79ace7e16ea5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:04:51.416871 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.416803 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-service-ca\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:04:51.417120 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.417095 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "53bd1832-fefd-4ea2-bd98-79ace7e16ea5" (UID: "53bd1832-fefd-4ea2-bd98-79ace7e16ea5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:04:51.417280 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.417128 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-config" (OuterVolumeSpecName: "console-config") pod "53bd1832-fefd-4ea2-bd98-79ace7e16ea5" (UID: "53bd1832-fefd-4ea2-bd98-79ace7e16ea5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:04:51.419392 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.419332 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "53bd1832-fefd-4ea2-bd98-79ace7e16ea5" (UID: "53bd1832-fefd-4ea2-bd98-79ace7e16ea5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:04:51.419502 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.419456 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "53bd1832-fefd-4ea2-bd98-79ace7e16ea5" (UID: "53bd1832-fefd-4ea2-bd98-79ace7e16ea5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:04:51.420207 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.420170 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-kube-api-access-vhxnb" (OuterVolumeSpecName: "kube-api-access-vhxnb") pod "53bd1832-fefd-4ea2-bd98-79ace7e16ea5" (UID: "53bd1832-fefd-4ea2-bd98-79ace7e16ea5"). InnerVolumeSpecName "kube-api-access-vhxnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:04:51.517116 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.517054 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:04:51.517116 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.517080 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-trusted-ca-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:04:51.517116 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.517095 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-oauth-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:04:51.517116 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.517108 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxnb\" (UniqueName: \"kubernetes.io/projected/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-kube-api-access-vhxnb\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:04:51.517320 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.517123 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:04:51.517320 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:51.517135 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bd1832-fefd-4ea2-bd98-79ace7e16ea5-console-oauth-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:04:52.093014 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:52.092949 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fccd774-5rk68" Jul 10 18:04:52.093014 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:52.092971 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fccd774-5rk68" event={"ID":"53bd1832-fefd-4ea2-bd98-79ace7e16ea5","Type":"ContainerDied","Data":"cc88e69d9033477657b6bd8bc829ae28ada8325a19bcc26f534a64bd2302617c"} Jul 10 18:04:52.093442 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:52.093029 2570 scope.go:117] "RemoveContainer" containerID="2ce3082b016e574f7b1ab8691022468fc7823992a2004672e78adcfcd58aefe8" Jul 10 18:04:52.108755 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:52.108733 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6fccd774-5rk68"] Jul 10 18:04:52.112487 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:52.112466 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f6fccd774-5rk68"] Jul 10 18:04:53.461740 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:04:53.461709 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53bd1832-fefd-4ea2-bd98-79ace7e16ea5" path="/var/lib/kubelet/pods/53bd1832-fefd-4ea2-bd98-79ace7e16ea5/volumes" Jul 10 18:05:04.283779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:04.283750 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:05:04.287685 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:04.287659 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-689b6dc5d-rnw2l" Jul 10 18:05:06.205172 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.205122 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c45bd8d8-kswn8" podUID="f2cbe0d2-c025-4b67-9c2c-36122525f8d0" containerName="console" containerID="cri-o://457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5" gracePeriod=15 Jul 10 18:05:06.438274 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.438249 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:05:06.523827 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.523754 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-oauth-config\") pod \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " Jul 10 18:05:06.523827 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.523786 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-serving-cert\") pod \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " Jul 10 18:05:06.523827 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.523809 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-oauth-serving-cert\") pod \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " Jul 10 18:05:06.523827 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.523827 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-kube-api-access-5l5qb\") pod \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " Jul 10 18:05:06.524088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.523862 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-service-ca\") pod \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " Jul 10 18:05:06.524088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.523885 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-config\") pod \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\" (UID: \"f2cbe0d2-c025-4b67-9c2c-36122525f8d0\") " Jul 10 18:05:06.524319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.524190 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f2cbe0d2-c025-4b67-9c2c-36122525f8d0" (UID: "f2cbe0d2-c025-4b67-9c2c-36122525f8d0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:05:06.524319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.524295 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-service-ca" (OuterVolumeSpecName: "service-ca") pod "f2cbe0d2-c025-4b67-9c2c-36122525f8d0" (UID: "f2cbe0d2-c025-4b67-9c2c-36122525f8d0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:05:06.524501 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.524367 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-config" (OuterVolumeSpecName: "console-config") pod "f2cbe0d2-c025-4b67-9c2c-36122525f8d0" (UID: "f2cbe0d2-c025-4b67-9c2c-36122525f8d0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:05:06.526082 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.526059 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f2cbe0d2-c025-4b67-9c2c-36122525f8d0" (UID: "f2cbe0d2-c025-4b67-9c2c-36122525f8d0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:05:06.526494 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.526473 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f2cbe0d2-c025-4b67-9c2c-36122525f8d0" (UID: "f2cbe0d2-c025-4b67-9c2c-36122525f8d0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:05:06.526543 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.526497 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-kube-api-access-5l5qb" (OuterVolumeSpecName: "kube-api-access-5l5qb") pod "f2cbe0d2-c025-4b67-9c2c-36122525f8d0" (UID: "f2cbe0d2-c025-4b67-9c2c-36122525f8d0"). InnerVolumeSpecName "kube-api-access-5l5qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:05:06.624905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.624865 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-oauth-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:05:06.624905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.624901 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:05:06.624905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.624911 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-oauth-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:05:06.625099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.624920 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-kube-api-access-5l5qb\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:05:06.625099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.624931 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-service-ca\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:05:06.625099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:06.624938 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2cbe0d2-c025-4b67-9c2c-36122525f8d0-console-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:05:07.154172 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.154142 2570 generic.go:356] "Generic (PLEG): container finished" podID="f2cbe0d2-c025-4b67-9c2c-36122525f8d0" containerID="457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5" exitCode=0 Jul 10 18:05:07.154287 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.154188 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45bd8d8-kswn8" Jul 10 18:05:07.154287 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.154218 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45bd8d8-kswn8" event={"ID":"f2cbe0d2-c025-4b67-9c2c-36122525f8d0","Type":"ContainerDied","Data":"457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5"} Jul 10 18:05:07.154287 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.154262 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45bd8d8-kswn8" event={"ID":"f2cbe0d2-c025-4b67-9c2c-36122525f8d0","Type":"ContainerDied","Data":"9353a8aa69a6597b725e27e97e8348586012588041d03e7ca659bbd7f9286a1f"} Jul 10 18:05:07.154287 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.154282 2570 scope.go:117] "RemoveContainer" containerID="457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5" Jul 10 18:05:07.164419 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.164101 2570 scope.go:117] "RemoveContainer" containerID="457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5" Jul 10 18:05:07.164615 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:05:07.164571 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5\": container with ID starting with 457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5 not found: ID does not exist" containerID="457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5" Jul 10 18:05:07.164778 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.164726 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5"} err="failed to get container status \"457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5\": rpc error: code = NotFound desc = could not find container \"457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5\": container with ID starting with 457096aecc3274f6d9b242a0a33c25d3f6182cb85de96eb3bdb5dd66d2a504c5 not found: ID does not exist" Jul 10 18:05:07.177856 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.177832 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c45bd8d8-kswn8"] Jul 10 18:05:07.181620 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.181583 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c45bd8d8-kswn8"] Jul 10 18:05:07.461679 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:07.461658 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cbe0d2-c025-4b67-9c2c-36122525f8d0" path="/var/lib/kubelet/pods/f2cbe0d2-c025-4b67-9c2c-36122525f8d0/volumes" Jul 10 18:05:53.879639 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.879611 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cdfb5b968-hbph7"] Jul 10 18:05:53.880055 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.879830 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bd1832-fefd-4ea2-bd98-79ace7e16ea5" containerName="console" Jul 10 18:05:53.880055 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.879841 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bd1832-fefd-4ea2-bd98-79ace7e16ea5" containerName="console" Jul 10 18:05:53.880055 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.879857 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cbe0d2-c025-4b67-9c2c-36122525f8d0" containerName="console" Jul 10 18:05:53.880055 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.879862 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cbe0d2-c025-4b67-9c2c-36122525f8d0" containerName="console" Jul 10 18:05:53.880055 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.879901 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cbe0d2-c025-4b67-9c2c-36122525f8d0" containerName="console" Jul 10 18:05:53.880055 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.879909 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="53bd1832-fefd-4ea2-bd98-79ace7e16ea5" containerName="console" Jul 10 18:05:53.881823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.881806 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:53.893503 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.893474 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdfb5b968-hbph7"] Jul 10 18:05:53.950320 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.950299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-serving-cert\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:53.950433 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.950329 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-oauth-serving-cert\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:53.950433 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.950356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-service-ca\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:53.950433 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.950381 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-oauth-config\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:53.950547 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.950447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-trusted-ca-bundle\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:53.950547 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.950473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqhnd\" (UniqueName: \"kubernetes.io/projected/7672036c-c69a-44e1-a8bf-04512ba8213a-kube-api-access-fqhnd\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:53.950547 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:53.950510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-console-config\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.051610 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.051582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-serving-cert\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.051707 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.051628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-oauth-serving-cert\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.051707 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.051649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-service-ca\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.051707 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.051671 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-oauth-config\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.051707 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.051697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-trusted-ca-bundle\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.051885 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.051722 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqhnd\" (UniqueName: \"kubernetes.io/projected/7672036c-c69a-44e1-a8bf-04512ba8213a-kube-api-access-fqhnd\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.051885 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.051767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-console-config\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.052386 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.052363 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-oauth-serving-cert\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.052548 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.052383 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-console-config\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.052684 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.052564 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-service-ca\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.052753 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.052732 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-trusted-ca-bundle\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.054203 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.054185 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-serving-cert\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.054203 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.054199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-oauth-config\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.060960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.060942 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqhnd\" (UniqueName: \"kubernetes.io/projected/7672036c-c69a-44e1-a8bf-04512ba8213a-kube-api-access-fqhnd\") pod \"console-7cdfb5b968-hbph7\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.191000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.190976 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:05:54.314088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:54.314035 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdfb5b968-hbph7"] Jul 10 18:05:54.316116 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:05:54.316091 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7672036c_c69a_44e1_a8bf_04512ba8213a.slice/crio-6d009fe872034e45b368329dd12031b26fba68e84ea67aa8392534a9899326e3 WatchSource:0}: Error finding container 6d009fe872034e45b368329dd12031b26fba68e84ea67aa8392534a9899326e3: Status 404 returned error can't find the container with id 6d009fe872034e45b368329dd12031b26fba68e84ea67aa8392534a9899326e3 Jul 10 18:05:55.312046 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:55.312017 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdfb5b968-hbph7" event={"ID":"7672036c-c69a-44e1-a8bf-04512ba8213a","Type":"ContainerStarted","Data":"f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199"} Jul 10 18:05:55.312372 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:55.312068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdfb5b968-hbph7" event={"ID":"7672036c-c69a-44e1-a8bf-04512ba8213a","Type":"ContainerStarted","Data":"6d009fe872034e45b368329dd12031b26fba68e84ea67aa8392534a9899326e3"} Jul 10 18:05:55.332401 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:05:55.332361 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cdfb5b968-hbph7" podStartSLOduration=2.332347408 podStartE2EDuration="2.332347408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:05:55.330802957 +0000 UTC m=+164.425993443" watchObservedRunningTime="2026-07-10 18:05:55.332347408 +0000 UTC m=+164.427537895" Jul 10 18:06:04.191472 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:04.191380 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:06:04.191472 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:04.191450 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:06:04.196325 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:04.196299 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:06:04.345723 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:04.345695 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:06:04.392309 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:04.392278 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d958895d-5ndmm"] Jul 10 18:06:29.412450 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.412410 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65d958895d-5ndmm" podUID="01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" containerName="console" containerID="cri-o://e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6" gracePeriod=15 Jul 10 18:06:29.649166 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.649144 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:06:29.719906 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.719822 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmtjz\" (UniqueName: \"kubernetes.io/projected/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-kube-api-access-pmtjz\") pod \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " Jul 10 18:06:29.719906 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.719860 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-serving-cert\") pod \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " Jul 10 18:06:29.719906 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.719888 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-oauth-config\") pod \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " Jul 10 18:06:29.720180 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.719912 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-config\") pod \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " Jul 10 18:06:29.720180 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.719937 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-service-ca\") pod \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " Jul 10 18:06:29.720180 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.720057 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-trusted-ca-bundle\") pod \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " Jul 10 18:06:29.720180 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.720132 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-oauth-serving-cert\") pod \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\" (UID: \"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f\") " Jul 10 18:06:29.720401 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.720374 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-service-ca" (OuterVolumeSpecName: "service-ca") pod "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" (UID: "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:06:29.720460 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.720417 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-config" (OuterVolumeSpecName: "console-config") pod "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" (UID: "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:06:29.720557 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.720458 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" (UID: "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:06:29.720804 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.720662 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" (UID: "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:06:29.722148 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.722127 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" (UID: "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:06:29.722612 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.722579 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" (UID: "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:06:29.722683 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.722630 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-kube-api-access-pmtjz" (OuterVolumeSpecName: "kube-api-access-pmtjz") pod "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" (UID: "01ec763d-a5c9-438c-9b2e-4ef63a55ba4f"). InnerVolumeSpecName "kube-api-access-pmtjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:06:29.821151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.821110 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtjz\" (UniqueName: \"kubernetes.io/projected/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-kube-api-access-pmtjz\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:06:29.821151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.821140 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:06:29.821151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.821154 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-oauth-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:06:29.821381 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.821164 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-console-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:06:29.821381 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.821174 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-service-ca\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:06:29.821381 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.821182 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-trusted-ca-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:06:29.821381 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:29.821190 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f-oauth-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:06:30.433049 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.433018 2570 generic.go:356] "Generic (PLEG): container finished" podID="01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" containerID="e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6" exitCode=0 Jul 10 18:06:30.433464 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.433069 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d958895d-5ndmm" Jul 10 18:06:30.433464 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.433113 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d958895d-5ndmm" event={"ID":"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f","Type":"ContainerDied","Data":"e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6"} Jul 10 18:06:30.433464 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.433166 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d958895d-5ndmm" event={"ID":"01ec763d-a5c9-438c-9b2e-4ef63a55ba4f","Type":"ContainerDied","Data":"4ac1bf282d5ea750f5ed5d7de3fca52f06f31f2cd0fdd5132454080cd447e53f"} Jul 10 18:06:30.433464 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.433186 2570 scope.go:117] "RemoveContainer" containerID="e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6" Jul 10 18:06:30.441512 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.441496 2570 scope.go:117] "RemoveContainer" containerID="e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6" Jul 10 18:06:30.441795 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:06:30.441778 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6\": container with ID starting with e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6 not found: ID does not exist" containerID="e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6" Jul 10 18:06:30.441846 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.441802 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6"} err="failed to get container status \"e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6\": rpc error: code = NotFound desc = could not find container \"e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6\": container with ID starting with e722335eb3e765de1e7fa4dc96e137ef25fb62a1f4acb88b3615265c4522b8c6 not found: ID does not exist" Jul 10 18:06:30.454047 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.454025 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d958895d-5ndmm"] Jul 10 18:06:30.458635 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:30.458615 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65d958895d-5ndmm"] Jul 10 18:06:31.466154 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:31.466119 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" path="/var/lib/kubelet/pods/01ec763d-a5c9-438c-9b2e-4ef63a55ba4f/volumes" Jul 10 18:06:59.759297 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.759253 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj"] Jul 10 18:06:59.759832 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.759581 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" containerName="console" Jul 10 18:06:59.759832 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.759613 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" containerName="console" Jul 10 18:06:59.759832 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.759668 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec763d-a5c9-438c-9b2e-4ef63a55ba4f" containerName="console" Jul 10 18:06:59.762366 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.762349 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.765017 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.764995 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 18:06:59.765143 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.764995 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 18:06:59.766020 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.766003 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ddmng\"" Jul 10 18:06:59.772017 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.771995 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj"] Jul 10 18:06:59.837214 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.837171 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hg7\" (UniqueName: \"kubernetes.io/projected/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-kube-api-access-w2hg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.837214 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.837212 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.837436 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.837235 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.938009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.937971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.938122 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.938060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hg7\" (UniqueName: \"kubernetes.io/projected/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-kube-api-access-w2hg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.938122 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.938084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.938374 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.938353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.938443 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.938396 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:06:59.948267 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:06:59.948243 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hg7\" (UniqueName: \"kubernetes.io/projected/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-kube-api-access-w2hg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:07:00.072494 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:00.072402 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:07:00.200085 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:00.199986 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj"] Jul 10 18:07:00.202730 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:07:00.202703 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc48dcd_76a3_4b83_a8ba_797f1041f7ab.slice/crio-4c03cde9f4046eb0b13a3740414889dd60e0b074155dd3265ca781e56f7b22e3 WatchSource:0}: Error finding container 4c03cde9f4046eb0b13a3740414889dd60e0b074155dd3265ca781e56f7b22e3: Status 404 returned error can't find the container with id 4c03cde9f4046eb0b13a3740414889dd60e0b074155dd3265ca781e56f7b22e3 Jul 10 18:07:00.536140 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:00.536095 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" event={"ID":"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab","Type":"ContainerStarted","Data":"4c03cde9f4046eb0b13a3740414889dd60e0b074155dd3265ca781e56f7b22e3"} Jul 10 18:07:05.557461 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:05.557361 2570 generic.go:356] "Generic (PLEG): container finished" podID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerID="a68ec95147652c2eda2b1ebf3bc31bf7815967f192bda8bd9c5d2b40de9a14fb" exitCode=0 Jul 10 18:07:05.557866 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:05.557448 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" event={"ID":"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab","Type":"ContainerDied","Data":"a68ec95147652c2eda2b1ebf3bc31bf7815967f192bda8bd9c5d2b40de9a14fb"} Jul 10 18:07:07.565651 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:07.565572 2570 generic.go:356] "Generic (PLEG): container finished" podID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerID="41a1a4d8be189fcadae04f03c47939aed1be590a47d8b63e61a448bbbd8eef11" exitCode=0 Jul 10 18:07:07.565651 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:07.565638 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" event={"ID":"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab","Type":"ContainerDied","Data":"41a1a4d8be189fcadae04f03c47939aed1be590a47d8b63e61a448bbbd8eef11"} Jul 10 18:07:15.597901 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:15.597857 2570 generic.go:356] "Generic (PLEG): container finished" podID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerID="7dcbd2f2b59b209f02b3f59a4538e2085b1e6a864d84469f16121d5af8b26bf1" exitCode=0 Jul 10 18:07:15.597901 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:15.597900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" event={"ID":"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab","Type":"ContainerDied","Data":"7dcbd2f2b59b209f02b3f59a4538e2085b1e6a864d84469f16121d5af8b26bf1"} Jul 10 18:07:16.726163 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.726137 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:07:16.785480 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.785453 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-util\") pod \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " Jul 10 18:07:16.785630 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.785534 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2hg7\" (UniqueName: \"kubernetes.io/projected/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-kube-api-access-w2hg7\") pod \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " Jul 10 18:07:16.785630 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.785554 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-bundle\") pod \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\" (UID: \"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab\") " Jul 10 18:07:16.786217 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.786197 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-bundle" (OuterVolumeSpecName: "bundle") pod "0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" (UID: "0fc48dcd-76a3-4b83-a8ba-797f1041f7ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:16.787750 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.787726 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-kube-api-access-w2hg7" (OuterVolumeSpecName: "kube-api-access-w2hg7") pod "0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" (UID: "0fc48dcd-76a3-4b83-a8ba-797f1041f7ab"). InnerVolumeSpecName "kube-api-access-w2hg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:16.789766 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.789745 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-util" (OuterVolumeSpecName: "util") pod "0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" (UID: "0fc48dcd-76a3-4b83-a8ba-797f1041f7ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:16.886191 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.886117 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hg7\" (UniqueName: \"kubernetes.io/projected/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-kube-api-access-w2hg7\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:16.886191 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.886147 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:16.886191 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:16.886159 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc48dcd-76a3-4b83-a8ba-797f1041f7ab-util\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:17.605460 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:17.605432 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" event={"ID":"0fc48dcd-76a3-4b83-a8ba-797f1041f7ab","Type":"ContainerDied","Data":"4c03cde9f4046eb0b13a3740414889dd60e0b074155dd3265ca781e56f7b22e3"} Jul 10 18:07:17.605629 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:17.605464 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qsztj" Jul 10 18:07:17.605629 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:17.605480 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c03cde9f4046eb0b13a3740414889dd60e0b074155dd3265ca781e56f7b22e3" Jul 10 18:07:23.337075 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337038 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst"] Jul 10 18:07:23.337477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337289 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerName="extract" Jul 10 18:07:23.337477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337300 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerName="extract" Jul 10 18:07:23.337477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337316 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerName="pull" Jul 10 18:07:23.337477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337320 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerName="pull" Jul 10 18:07:23.337477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337332 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerName="util" Jul 10 18:07:23.337477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337338 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerName="util" Jul 10 18:07:23.337477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.337375 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc48dcd-76a3-4b83-a8ba-797f1041f7ab" containerName="extract" Jul 10 18:07:23.376562 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.376524 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst"] Jul 10 18:07:23.376742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.376644 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.381538 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.381515 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:07:23.381703 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.381669 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8sqpf\"" Jul 10 18:07:23.382198 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.382180 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 18:07:23.430842 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.430808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a0139f9-b2f1-4e59-b507-07e3c66b0344-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j2lst\" (UID: \"7a0139f9-b2f1-4e59-b507-07e3c66b0344\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.431000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.430892 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kc4d\" (UniqueName: \"kubernetes.io/projected/7a0139f9-b2f1-4e59-b507-07e3c66b0344-kube-api-access-9kc4d\") pod \"cert-manager-operator-controller-manager-55c444bf86-j2lst\" (UID: \"7a0139f9-b2f1-4e59-b507-07e3c66b0344\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.531373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.531340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kc4d\" (UniqueName: \"kubernetes.io/projected/7a0139f9-b2f1-4e59-b507-07e3c66b0344-kube-api-access-9kc4d\") pod \"cert-manager-operator-controller-manager-55c444bf86-j2lst\" (UID: \"7a0139f9-b2f1-4e59-b507-07e3c66b0344\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.531514 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.531390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a0139f9-b2f1-4e59-b507-07e3c66b0344-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j2lst\" (UID: \"7a0139f9-b2f1-4e59-b507-07e3c66b0344\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.531717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.531702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a0139f9-b2f1-4e59-b507-07e3c66b0344-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j2lst\" (UID: \"7a0139f9-b2f1-4e59-b507-07e3c66b0344\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.547131 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.547102 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kc4d\" (UniqueName: \"kubernetes.io/projected/7a0139f9-b2f1-4e59-b507-07e3c66b0344-kube-api-access-9kc4d\") pod \"cert-manager-operator-controller-manager-55c444bf86-j2lst\" (UID: \"7a0139f9-b2f1-4e59-b507-07e3c66b0344\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.686013 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.685988 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" Jul 10 18:07:23.837793 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:23.837762 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst"] Jul 10 18:07:23.843418 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:07:23.843393 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0139f9_b2f1_4e59_b507_07e3c66b0344.slice/crio-b34667ad33eb9e96afe8e14516b6a3d91c8c77d151ee566cd7a18054f5e4c9d7 WatchSource:0}: Error finding container b34667ad33eb9e96afe8e14516b6a3d91c8c77d151ee566cd7a18054f5e4c9d7: Status 404 returned error can't find the container with id b34667ad33eb9e96afe8e14516b6a3d91c8c77d151ee566cd7a18054f5e4c9d7 Jul 10 18:07:24.633675 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:24.633636 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" event={"ID":"7a0139f9-b2f1-4e59-b507-07e3c66b0344","Type":"ContainerStarted","Data":"b34667ad33eb9e96afe8e14516b6a3d91c8c77d151ee566cd7a18054f5e4c9d7"} Jul 10 18:07:26.642477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:26.642443 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" event={"ID":"7a0139f9-b2f1-4e59-b507-07e3c66b0344","Type":"ContainerStarted","Data":"8db66cc838ba9d942b5c3c67421a498d4d0387da6f1df0bd9a21bd8935a9a5db"} Jul 10 18:07:26.667528 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:26.667476 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j2lst" podStartSLOduration=1.425936821 podStartE2EDuration="3.667461692s" totalImagesPullingTime="2.241524871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:07:26.665235219 +0000 UTC m=+255.760425716" watchObservedRunningTime="2026-07-10 18:07:26.667461692 +0000 UTC m=+255.762652177" Jul 10 18:07:27.411679 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.411639 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp"] Jul 10 18:07:27.431678 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.431653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.435088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.435045 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp"] Jul 10 18:07:27.436791 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.436767 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 18:07:27.436914 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.436827 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ddmng\"" Jul 10 18:07:27.436914 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.436827 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 18:07:27.460538 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.460516 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.460676 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.460549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m99tl\" (UniqueName: \"kubernetes.io/projected/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-kube-api-access-m99tl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.460676 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.460593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.561840 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.561815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m99tl\" (UniqueName: \"kubernetes.io/projected/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-kube-api-access-m99tl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.561982 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.561857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.561982 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.561924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.562233 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.562214 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.562302 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.562243 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.580895 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.580870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m99tl\" (UniqueName: \"kubernetes.io/projected/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-kube-api-access-m99tl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.742965 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.742895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:27.922304 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:27.922259 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp"] Jul 10 18:07:27.925065 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:07:27.925036 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc710571b_99e0_4a5c_8d4b_6d12eb1d1fcf.slice/crio-33b57c781d7d593b0fbd7a849e3b2e3dac5f30ae9e8512733a8fa392f1516bbb WatchSource:0}: Error finding container 33b57c781d7d593b0fbd7a849e3b2e3dac5f30ae9e8512733a8fa392f1516bbb: Status 404 returned error can't find the container with id 33b57c781d7d593b0fbd7a849e3b2e3dac5f30ae9e8512733a8fa392f1516bbb Jul 10 18:07:28.650503 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:28.650470 2570 generic.go:356] "Generic (PLEG): container finished" podID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerID="54d61442050455027eb667213d84acd8b71daa76184db9cd1082a7b9407528c0" exitCode=0 Jul 10 18:07:28.650706 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:28.650562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" event={"ID":"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf","Type":"ContainerDied","Data":"54d61442050455027eb667213d84acd8b71daa76184db9cd1082a7b9407528c0"} Jul 10 18:07:28.650706 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:28.650635 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" event={"ID":"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf","Type":"ContainerStarted","Data":"33b57c781d7d593b0fbd7a849e3b2e3dac5f30ae9e8512733a8fa392f1516bbb"} Jul 10 18:07:29.806682 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.806637 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr"] Jul 10 18:07:29.808939 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.808919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:29.812112 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.811978 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 18:07:29.812112 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.811994 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tmvz8\"" Jul 10 18:07:29.813079 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.813061 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 18:07:29.824059 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.824033 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr"] Jul 10 18:07:29.880568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.880533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861ce5e6-bb8b-4590-9689-9d0bb3d2e860-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgcpr\" (UID: \"861ce5e6-bb8b-4590-9689-9d0bb3d2e860\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:29.880746 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.880657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96jmd\" (UniqueName: \"kubernetes.io/projected/861ce5e6-bb8b-4590-9689-9d0bb3d2e860-kube-api-access-96jmd\") pod \"cert-manager-cainjector-559cdd8c87-bgcpr\" (UID: \"861ce5e6-bb8b-4590-9689-9d0bb3d2e860\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:29.981632 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.981578 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96jmd\" (UniqueName: \"kubernetes.io/projected/861ce5e6-bb8b-4590-9689-9d0bb3d2e860-kube-api-access-96jmd\") pod \"cert-manager-cainjector-559cdd8c87-bgcpr\" (UID: \"861ce5e6-bb8b-4590-9689-9d0bb3d2e860\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:29.981804 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.981685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861ce5e6-bb8b-4590-9689-9d0bb3d2e860-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgcpr\" (UID: \"861ce5e6-bb8b-4590-9689-9d0bb3d2e860\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:29.990750 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.990725 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861ce5e6-bb8b-4590-9689-9d0bb3d2e860-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgcpr\" (UID: \"861ce5e6-bb8b-4590-9689-9d0bb3d2e860\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:29.990942 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:29.990918 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96jmd\" (UniqueName: \"kubernetes.io/projected/861ce5e6-bb8b-4590-9689-9d0bb3d2e860-kube-api-access-96jmd\") pod \"cert-manager-cainjector-559cdd8c87-bgcpr\" (UID: \"861ce5e6-bb8b-4590-9689-9d0bb3d2e860\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:30.129913 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:30.129819 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" Jul 10 18:07:30.764313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:30.764213 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr"] Jul 10 18:07:30.767047 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:07:30.767016 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod861ce5e6_bb8b_4590_9689_9d0bb3d2e860.slice/crio-b147b810a39114eb4ef2d81aa60692334a3e9969b2b9b54dd91282777033e88d WatchSource:0}: Error finding container b147b810a39114eb4ef2d81aa60692334a3e9969b2b9b54dd91282777033e88d: Status 404 returned error can't find the container with id b147b810a39114eb4ef2d81aa60692334a3e9969b2b9b54dd91282777033e88d Jul 10 18:07:31.667251 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:31.667156 2570 generic.go:356] "Generic (PLEG): container finished" podID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerID="fc7dfc571a18fc2307c1d477399dd4b9ce93a1166f9b017fd70b1d924ff1c477" exitCode=0 Jul 10 18:07:31.667765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:31.667249 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" event={"ID":"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf","Type":"ContainerDied","Data":"fc7dfc571a18fc2307c1d477399dd4b9ce93a1166f9b017fd70b1d924ff1c477"} Jul 10 18:07:31.668664 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:31.668638 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" event={"ID":"861ce5e6-bb8b-4590-9689-9d0bb3d2e860","Type":"ContainerStarted","Data":"b147b810a39114eb4ef2d81aa60692334a3e9969b2b9b54dd91282777033e88d"} Jul 10 18:07:32.676158 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:32.676119 2570 generic.go:356] "Generic (PLEG): container finished" podID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerID="35eb4cdfb8cf52683a34088c44ac291f72e172b0087572812bac12dc2a4716d1" exitCode=0 Jul 10 18:07:32.676570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:32.676200 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" event={"ID":"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf","Type":"ContainerDied","Data":"35eb4cdfb8cf52683a34088c44ac291f72e172b0087572812bac12dc2a4716d1"} Jul 10 18:07:33.682070 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.682030 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" event={"ID":"861ce5e6-bb8b-4590-9689-9d0bb3d2e860","Type":"ContainerStarted","Data":"36af48223feaebfc714413a1e6600f8e07c939521472bee934317d922511949b"} Jul 10 18:07:33.701971 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.701911 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgcpr" podStartSLOduration=2.190242784 podStartE2EDuration="4.701887586s" totalImagesPullingTime="2.511644802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:07:33.69983372 +0000 UTC m=+262.795024206" watchObservedRunningTime="2026-07-10 18:07:33.701887586 +0000 UTC m=+262.797078073" Jul 10 18:07:33.817285 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.817260 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:33.915216 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.915181 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-bundle\") pod \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " Jul 10 18:07:33.915410 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.915236 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-util\") pod \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " Jul 10 18:07:33.915410 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.915275 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m99tl\" (UniqueName: \"kubernetes.io/projected/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-kube-api-access-m99tl\") pod \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\" (UID: \"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf\") " Jul 10 18:07:33.915545 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.915523 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-bundle" (OuterVolumeSpecName: "bundle") pod "c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" (UID: "c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:33.917501 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.917476 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-kube-api-access-m99tl" (OuterVolumeSpecName: "kube-api-access-m99tl") pod "c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" (UID: "c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf"). InnerVolumeSpecName "kube-api-access-m99tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:33.919789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:33.919758 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-util" (OuterVolumeSpecName: "util") pod "c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" (UID: "c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:34.016045 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.015948 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:34.016045 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.015989 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-util\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:34.016045 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.015999 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m99tl\" (UniqueName: \"kubernetes.io/projected/c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf-kube-api-access-m99tl\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:34.687387 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.687358 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" Jul 10 18:07:34.687387 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.687380 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnlhtp" event={"ID":"c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf","Type":"ContainerDied","Data":"33b57c781d7d593b0fbd7a849e3b2e3dac5f30ae9e8512733a8fa392f1516bbb"} Jul 10 18:07:34.687821 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.687422 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b57c781d7d593b0fbd7a849e3b2e3dac5f30ae9e8512733a8fa392f1516bbb" Jul 10 18:07:34.752028 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.751986 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5pbmh"] Jul 10 18:07:34.752323 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.752306 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerName="util" Jul 10 18:07:34.752396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.752327 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerName="util" Jul 10 18:07:34.752396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.752342 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerName="pull" Jul 10 18:07:34.752396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.752352 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerName="pull" Jul 10 18:07:34.752396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.752366 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerName="extract" Jul 10 18:07:34.752396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.752374 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerName="extract" Jul 10 18:07:34.752655 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.752447 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c710571b-99e0-4a5c-8d4b-6d12eb1d1fcf" containerName="extract" Jul 10 18:07:34.754490 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.754467 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:34.757380 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.757361 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lv7zr\"" Jul 10 18:07:34.764573 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.764551 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5pbmh"] Jul 10 18:07:34.823467 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.823432 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a31b888e-d0e2-4c0b-ab82-1f005e48f3b7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5pbmh\" (UID: \"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:34.823676 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.823486 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7pj\" (UniqueName: \"kubernetes.io/projected/a31b888e-d0e2-4c0b-ab82-1f005e48f3b7-kube-api-access-tp7pj\") pod \"cert-manager-webhook-6486ffd997-5pbmh\" (UID: \"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:34.924674 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.924635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a31b888e-d0e2-4c0b-ab82-1f005e48f3b7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5pbmh\" (UID: \"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:34.924874 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.924690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7pj\" (UniqueName: \"kubernetes.io/projected/a31b888e-d0e2-4c0b-ab82-1f005e48f3b7-kube-api-access-tp7pj\") pod \"cert-manager-webhook-6486ffd997-5pbmh\" (UID: \"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:34.934326 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.934293 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a31b888e-d0e2-4c0b-ab82-1f005e48f3b7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5pbmh\" (UID: \"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:34.934437 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:34.934397 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7pj\" (UniqueName: \"kubernetes.io/projected/a31b888e-d0e2-4c0b-ab82-1f005e48f3b7-kube-api-access-tp7pj\") pod \"cert-manager-webhook-6486ffd997-5pbmh\" (UID: \"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:35.063955 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:35.063866 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:35.199099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:35.198885 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5pbmh"] Jul 10 18:07:35.201814 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:07:35.201787 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda31b888e_d0e2_4c0b_ab82_1f005e48f3b7.slice/crio-2df8bdfbf4b7e36dc5ff2efb965f6b1141d39c3f97ec04ce3e2f8d8c357a3a99 WatchSource:0}: Error finding container 2df8bdfbf4b7e36dc5ff2efb965f6b1141d39c3f97ec04ce3e2f8d8c357a3a99: Status 404 returned error can't find the container with id 2df8bdfbf4b7e36dc5ff2efb965f6b1141d39c3f97ec04ce3e2f8d8c357a3a99 Jul 10 18:07:35.697799 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:35.697754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" event={"ID":"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7","Type":"ContainerStarted","Data":"73c1820bade98a4480ab18908e856bbdd97a4fe3062973cbc633f1ed9efb6f32"} Jul 10 18:07:35.697799 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:35.697803 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" event={"ID":"a31b888e-d0e2-4c0b-ab82-1f005e48f3b7","Type":"ContainerStarted","Data":"2df8bdfbf4b7e36dc5ff2efb965f6b1141d39c3f97ec04ce3e2f8d8c357a3a99"} Jul 10 18:07:35.698221 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:35.697858 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:41.704207 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:41.704174 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" Jul 10 18:07:41.723990 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:41.723938 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5pbmh" podStartSLOduration=7.723917567 podStartE2EDuration="7.723917567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:07:35.734476889 +0000 UTC m=+264.829667375" watchObservedRunningTime="2026-07-10 18:07:41.723917567 +0000 UTC m=+270.819108053" Jul 10 18:07:42.931860 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.931820 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z"] Jul 10 18:07:42.934773 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.934750 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:42.937497 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.937475 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:07:42.938517 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.938501 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 18:07:42.938584 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.938506 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-l72lc\"" Jul 10 18:07:42.944716 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.944690 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z"] Jul 10 18:07:42.988905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.988848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6x9\" (UniqueName: \"kubernetes.io/projected/c89a2afe-fdef-465e-91fe-bc0351080d9b-kube-api-access-9b6x9\") pod \"openshift-lws-operator-fd8ccff4c-pnm5z\" (UID: \"c89a2afe-fdef-465e-91fe-bc0351080d9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:42.988905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:42.988909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89a2afe-fdef-465e-91fe-bc0351080d9b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pnm5z\" (UID: \"c89a2afe-fdef-465e-91fe-bc0351080d9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:43.089323 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:43.089284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6x9\" (UniqueName: \"kubernetes.io/projected/c89a2afe-fdef-465e-91fe-bc0351080d9b-kube-api-access-9b6x9\") pod \"openshift-lws-operator-fd8ccff4c-pnm5z\" (UID: \"c89a2afe-fdef-465e-91fe-bc0351080d9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:43.089502 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:43.089335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89a2afe-fdef-465e-91fe-bc0351080d9b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pnm5z\" (UID: \"c89a2afe-fdef-465e-91fe-bc0351080d9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:43.089789 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:43.089771 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89a2afe-fdef-465e-91fe-bc0351080d9b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pnm5z\" (UID: \"c89a2afe-fdef-465e-91fe-bc0351080d9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:43.098968 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:43.098944 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6x9\" (UniqueName: \"kubernetes.io/projected/c89a2afe-fdef-465e-91fe-bc0351080d9b-kube-api-access-9b6x9\") pod \"openshift-lws-operator-fd8ccff4c-pnm5z\" (UID: \"c89a2afe-fdef-465e-91fe-bc0351080d9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:43.244414 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:43.244323 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" Jul 10 18:07:43.402178 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:43.402151 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z"] Jul 10 18:07:43.404879 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:07:43.404850 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc89a2afe_fdef_465e_91fe_bc0351080d9b.slice/crio-df0808b737255048a1caedd9767ee01e79bf574ca3d3450bcfce4d48e0395242 WatchSource:0}: Error finding container df0808b737255048a1caedd9767ee01e79bf574ca3d3450bcfce4d48e0395242: Status 404 returned error can't find the container with id df0808b737255048a1caedd9767ee01e79bf574ca3d3450bcfce4d48e0395242 Jul 10 18:07:43.729003 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:43.728965 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" event={"ID":"c89a2afe-fdef-465e-91fe-bc0351080d9b","Type":"ContainerStarted","Data":"df0808b737255048a1caedd9767ee01e79bf574ca3d3450bcfce4d48e0395242"} Jul 10 18:07:46.743938 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:46.743874 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" event={"ID":"c89a2afe-fdef-465e-91fe-bc0351080d9b","Type":"ContainerStarted","Data":"69f8e526c08118092f041f2d89b5cf4fc7b7d1a95704ace54fabeec519e97cda"} Jul 10 18:07:46.777458 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:46.777366 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnm5z" podStartSLOduration=2.46600078 podStartE2EDuration="4.77733798s" totalImagesPullingTime="2.3113372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:07:46.775677245 +0000 UTC m=+275.870867729" watchObservedRunningTime="2026-07-10 18:07:46.77733798 +0000 UTC m=+275.872528465" Jul 10 18:07:49.345810 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.345779 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb"] Jul 10 18:07:49.348157 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.348139 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.351203 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.351179 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 18:07:49.351203 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.351213 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 18:07:49.352532 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.352511 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ddmng\"" Jul 10 18:07:49.368481 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.368449 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb"] Jul 10 18:07:49.443447 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.443402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7sx\" (UniqueName: \"kubernetes.io/projected/7e8159ca-5b56-4270-91ab-01fabd4478f2-kube-api-access-lr7sx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.443678 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.443466 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.443678 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.443536 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.544513 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.544474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7sx\" (UniqueName: \"kubernetes.io/projected/7e8159ca-5b56-4270-91ab-01fabd4478f2-kube-api-access-lr7sx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.544513 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.544529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.544799 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.544585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.545042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.545021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.545118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.545028 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.555995 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.555962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7sx\" (UniqueName: \"kubernetes.io/projected/7e8159ca-5b56-4270-91ab-01fabd4478f2-kube-api-access-lr7sx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.657449 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.657361 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:49.800094 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:49.800034 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb"] Jul 10 18:07:49.802708 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:07:49.802680 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8159ca_5b56_4270_91ab_01fabd4478f2.slice/crio-314ac758d0c8ed836f31853ab6753f87ebfa8534ae61c642ed0510228e40da6b WatchSource:0}: Error finding container 314ac758d0c8ed836f31853ab6753f87ebfa8534ae61c642ed0510228e40da6b: Status 404 returned error can't find the container with id 314ac758d0c8ed836f31853ab6753f87ebfa8534ae61c642ed0510228e40da6b Jul 10 18:07:50.763326 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:50.763292 2570 generic.go:356] "Generic (PLEG): container finished" podID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerID="062689ba6b045ea85f95268fbfe4446dd54ecc4dd5cb0ec155c2833efaa9c0cc" exitCode=0 Jul 10 18:07:50.763715 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:50.763352 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" event={"ID":"7e8159ca-5b56-4270-91ab-01fabd4478f2","Type":"ContainerDied","Data":"062689ba6b045ea85f95268fbfe4446dd54ecc4dd5cb0ec155c2833efaa9c0cc"} Jul 10 18:07:50.763715 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:50.763395 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" event={"ID":"7e8159ca-5b56-4270-91ab-01fabd4478f2","Type":"ContainerStarted","Data":"314ac758d0c8ed836f31853ab6753f87ebfa8534ae61c642ed0510228e40da6b"} Jul 10 18:07:51.768998 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:51.768959 2570 generic.go:356] "Generic (PLEG): container finished" podID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerID="96889736c59da7cbb0e74e2ff07b3b2f4d81e2c2d23e76701f30ba27c40db71f" exitCode=0 Jul 10 18:07:51.769402 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:51.769042 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" event={"ID":"7e8159ca-5b56-4270-91ab-01fabd4478f2","Type":"ContainerDied","Data":"96889736c59da7cbb0e74e2ff07b3b2f4d81e2c2d23e76701f30ba27c40db71f"} Jul 10 18:07:52.775673 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:52.775630 2570 generic.go:356] "Generic (PLEG): container finished" podID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerID="c9d9588609f8f8ca9545992e2b54c224c963fffa55754c7096a97c21dac97bd6" exitCode=0 Jul 10 18:07:52.776041 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:52.775719 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" event={"ID":"7e8159ca-5b56-4270-91ab-01fabd4478f2","Type":"ContainerDied","Data":"c9d9588609f8f8ca9545992e2b54c224c963fffa55754c7096a97c21dac97bd6"} Jul 10 18:07:53.903252 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:53.903222 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:53.976029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:53.975975 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-util\") pod \"7e8159ca-5b56-4270-91ab-01fabd4478f2\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " Jul 10 18:07:53.976208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:53.976051 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-bundle\") pod \"7e8159ca-5b56-4270-91ab-01fabd4478f2\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " Jul 10 18:07:53.976208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:53.976084 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr7sx\" (UniqueName: \"kubernetes.io/projected/7e8159ca-5b56-4270-91ab-01fabd4478f2-kube-api-access-lr7sx\") pod \"7e8159ca-5b56-4270-91ab-01fabd4478f2\" (UID: \"7e8159ca-5b56-4270-91ab-01fabd4478f2\") " Jul 10 18:07:53.976829 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:53.976801 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-bundle" (OuterVolumeSpecName: "bundle") pod "7e8159ca-5b56-4270-91ab-01fabd4478f2" (UID: "7e8159ca-5b56-4270-91ab-01fabd4478f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:53.978329 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:53.978294 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8159ca-5b56-4270-91ab-01fabd4478f2-kube-api-access-lr7sx" (OuterVolumeSpecName: "kube-api-access-lr7sx") pod "7e8159ca-5b56-4270-91ab-01fabd4478f2" (UID: "7e8159ca-5b56-4270-91ab-01fabd4478f2"). InnerVolumeSpecName "kube-api-access-lr7sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:53.981569 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:53.981536 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-util" (OuterVolumeSpecName: "util") pod "7e8159ca-5b56-4270-91ab-01fabd4478f2" (UID: "7e8159ca-5b56-4270-91ab-01fabd4478f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:54.076938 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:54.076835 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-util\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:54.076938 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:54.076880 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8159ca-5b56-4270-91ab-01fabd4478f2-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:54.076938 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:54.076891 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr7sx\" (UniqueName: \"kubernetes.io/projected/7e8159ca-5b56-4270-91ab-01fabd4478f2-kube-api-access-lr7sx\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:07:54.785930 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:54.785903 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" Jul 10 18:07:54.786098 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:54.785899 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afllnwb" event={"ID":"7e8159ca-5b56-4270-91ab-01fabd4478f2","Type":"ContainerDied","Data":"314ac758d0c8ed836f31853ab6753f87ebfa8534ae61c642ed0510228e40da6b"} Jul 10 18:07:54.786098 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:07:54.786026 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="314ac758d0c8ed836f31853ab6753f87ebfa8534ae61c642ed0510228e40da6b" Jul 10 18:08:06.009035 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.008996 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8"] Jul 10 18:08:06.009556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.009263 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerName="extract" Jul 10 18:08:06.009556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.009274 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerName="extract" Jul 10 18:08:06.009556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.009284 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerName="pull" Jul 10 18:08:06.009556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.009289 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerName="pull" Jul 10 18:08:06.009556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.009299 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerName="util" Jul 10 18:08:06.009556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.009305 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerName="util" Jul 10 18:08:06.009556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.009355 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8159ca-5b56-4270-91ab-01fabd4478f2" containerName="extract" Jul 10 18:08:06.045372 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.045312 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8"] Jul 10 18:08:06.045572 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.045479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.048441 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.048421 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gvddb\"" Jul 10 18:08:06.049663 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.049646 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 18:08:06.049851 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.049831 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 18:08:06.050048 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.050034 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 18:08:06.050146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.050124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 18:08:06.182433 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.182391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52a7d721-1131-4935-909b-043dd4f03e57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.182636 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.182446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpj6\" (UniqueName: \"kubernetes.io/projected/52a7d721-1131-4935-909b-043dd4f03e57-kube-api-access-jcpj6\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.182636 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.182516 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a7d721-1131-4935-909b-043dd4f03e57-webhook-cert\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.182636 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.182621 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52a7d721-1131-4935-909b-043dd4f03e57-manifests\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.182744 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.182646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a7d721-1131-4935-909b-043dd4f03e57-tmp\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.283440 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.283345 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52a7d721-1131-4935-909b-043dd4f03e57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.283440 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.283405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpj6\" (UniqueName: \"kubernetes.io/projected/52a7d721-1131-4935-909b-043dd4f03e57-kube-api-access-jcpj6\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.283440 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.283425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a7d721-1131-4935-909b-043dd4f03e57-webhook-cert\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.283714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.283471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52a7d721-1131-4935-909b-043dd4f03e57-manifests\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.283714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.283490 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a7d721-1131-4935-909b-043dd4f03e57-tmp\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.283977 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.283957 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52a7d721-1131-4935-909b-043dd4f03e57-manifests\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.286089 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.286065 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a7d721-1131-4935-909b-043dd4f03e57-tmp\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.286220 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.286126 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52a7d721-1131-4935-909b-043dd4f03e57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.286380 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.286360 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a7d721-1131-4935-909b-043dd4f03e57-webhook-cert\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.292848 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.292785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpj6\" (UniqueName: \"kubernetes.io/projected/52a7d721-1131-4935-909b-043dd4f03e57-kube-api-access-jcpj6\") pod \"opendatahub-operator-controller-manager-795b486df7-4bvw8\" (UID: \"52a7d721-1131-4935-909b-043dd4f03e57\") " pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.356731 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.356700 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:06.513764 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.513736 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8"] Jul 10 18:08:06.516550 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:08:06.516519 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a7d721_1131_4935_909b_043dd4f03e57.slice/crio-fe4a018c21152512462a71680ccc375b965642df407836f9d9685cffd8aece25 WatchSource:0}: Error finding container fe4a018c21152512462a71680ccc375b965642df407836f9d9685cffd8aece25: Status 404 returned error can't find the container with id fe4a018c21152512462a71680ccc375b965642df407836f9d9685cffd8aece25 Jul 10 18:08:06.840248 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:06.840214 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" event={"ID":"52a7d721-1131-4935-909b-043dd4f03e57","Type":"ContainerStarted","Data":"fe4a018c21152512462a71680ccc375b965642df407836f9d9685cffd8aece25"} Jul 10 18:08:09.858039 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:09.858000 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" event={"ID":"52a7d721-1131-4935-909b-043dd4f03e57","Type":"ContainerStarted","Data":"493e46e6fcae8b54f9d7b62e065e039be15a752a1bdca7bcbf40b91473b69b49"} Jul 10 18:08:10.863657 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:10.863621 2570 generic.go:356] "Generic (PLEG): container finished" podID="52a7d721-1131-4935-909b-043dd4f03e57" containerID="493e46e6fcae8b54f9d7b62e065e039be15a752a1bdca7bcbf40b91473b69b49" exitCode=0 Jul 10 18:08:10.863657 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:10.863658 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" event={"ID":"52a7d721-1131-4935-909b-043dd4f03e57","Type":"ContainerDied","Data":"493e46e6fcae8b54f9d7b62e065e039be15a752a1bdca7bcbf40b91473b69b49"} Jul 10 18:08:11.334397 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:11.334358 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:08:11.334570 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:11.334490 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:08:11.338887 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:11.338871 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 18:08:11.872040 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:11.871997 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" event={"ID":"52a7d721-1131-4935-909b-043dd4f03e57","Type":"ContainerStarted","Data":"41d34724fec1de5c9b75d002d828c30887da65299bfaae01e38b12afe3f7e542"} Jul 10 18:08:11.874339 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:11.872129 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:11.927449 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:11.927401 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" podStartSLOduration=2.370730363 podStartE2EDuration="6.927385425s" totalImagesPullingTime="3.464704732s" totalInitContainerRuntime="1.09195033s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:08:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:08:11.926711701 +0000 UTC m=+301.021902186" watchObservedRunningTime="2026-07-10 18:08:11.927385425 +0000 UTC m=+301.022575911" Jul 10 18:08:22.878920 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:22.878892 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-795b486df7-4bvw8" Jul 10 18:08:25.995545 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:25.995506 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-655946c88b-vf5p4"] Jul 10 18:08:26.001877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.001844 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.026160 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.026124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 18:08:26.026435 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.026415 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 18:08:26.027396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.027380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 18:08:26.040190 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.040161 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jjkl5\"" Jul 10 18:08:26.040299 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.040271 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 18:08:26.044804 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.044782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-tls-certs\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.044905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.044846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96qcj\" (UniqueName: \"kubernetes.io/projected/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-kube-api-access-96qcj\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.044905 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.044869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-tmp\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.073368 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.073319 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-655946c88b-vf5p4"] Jul 10 18:08:26.122443 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.122401 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ps68r"] Jul 10 18:08:26.125113 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.125088 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:26.130273 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.130254 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zp8pg\"" Jul 10 18:08:26.136829 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.136811 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 18:08:26.145928 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.145905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-tmp\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.146008 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.145941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkxb\" (UniqueName: \"kubernetes.io/projected/a3762dab-7ad1-440e-bbff-d8a6df59eccf-kube-api-access-9fkxb\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:26.146008 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.145968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:26.146078 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.146030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-tls-certs\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.146132 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.146114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96qcj\" (UniqueName: \"kubernetes.io/projected/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-kube-api-access-96qcj\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.148375 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.148356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-tmp\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.148610 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.148578 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-tls-certs\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.155899 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.155871 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ps68r"] Jul 10 18:08:26.178000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.177964 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96qcj\" (UniqueName: \"kubernetes.io/projected/dd0ff8f4-884b-4102-bd8a-409bad35e7b6-kube-api-access-96qcj\") pod \"kube-auth-proxy-655946c88b-vf5p4\" (UID: \"dd0ff8f4-884b-4102-bd8a-409bad35e7b6\") " pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.247451 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.247362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkxb\" (UniqueName: \"kubernetes.io/projected/a3762dab-7ad1-440e-bbff-d8a6df59eccf-kube-api-access-9fkxb\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:26.247451 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.247402 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:26.247670 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:08:26.247498 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 18:08:26.247670 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:08:26.247558 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert podName:a3762dab-7ad1-440e-bbff-d8a6df59eccf nodeName:}" failed. No retries permitted until 2026-07-10 18:08:26.747542272 +0000 UTC m=+315.842732736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert") pod "odh-model-controller-7998c67dc4-ps68r" (UID: "a3762dab-7ad1-440e-bbff-d8a6df59eccf") : secret "odh-model-controller-webhook-cert" not found Jul 10 18:08:26.269371 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.269338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkxb\" (UniqueName: \"kubernetes.io/projected/a3762dab-7ad1-440e-bbff-d8a6df59eccf-kube-api-access-9fkxb\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:26.313235 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.313191 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" Jul 10 18:08:26.460265 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.460239 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-655946c88b-vf5p4"] Jul 10 18:08:26.460985 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:08:26.460962 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0ff8f4_884b_4102_bd8a_409bad35e7b6.slice/crio-9e97cee968b52546559cb853e3ec8dc481d352e52ff1cbe3d2688d2b8536aea3 WatchSource:0}: Error finding container 9e97cee968b52546559cb853e3ec8dc481d352e52ff1cbe3d2688d2b8536aea3: Status 404 returned error can't find the container with id 9e97cee968b52546559cb853e3ec8dc481d352e52ff1cbe3d2688d2b8536aea3 Jul 10 18:08:26.462720 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.462706 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:08:26.751976 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.751941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:26.752160 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:08:26.752112 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 18:08:26.752198 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:08:26.752176 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert podName:a3762dab-7ad1-440e-bbff-d8a6df59eccf nodeName:}" failed. No retries permitted until 2026-07-10 18:08:27.752159318 +0000 UTC m=+316.847349783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert") pod "odh-model-controller-7998c67dc4-ps68r" (UID: "a3762dab-7ad1-440e-bbff-d8a6df59eccf") : secret "odh-model-controller-webhook-cert" not found Jul 10 18:08:26.934738 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.934701 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" event={"ID":"dd0ff8f4-884b-4102-bd8a-409bad35e7b6","Type":"ContainerStarted","Data":"9e97cee968b52546559cb853e3ec8dc481d352e52ff1cbe3d2688d2b8536aea3"} Jul 10 18:08:26.943565 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.943533 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4jb82"] Jul 10 18:08:26.946484 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.946465 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:26.950931 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.950912 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 18:08:26.953712 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.953692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wh89\" (UniqueName: \"kubernetes.io/projected/1a43527f-276f-4d39-bf1d-3146106459a6-kube-api-access-7wh89\") pod \"model-serving-api-54d89bd5fc-4jb82\" (UID: \"1a43527f-276f-4d39-bf1d-3146106459a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:26.953839 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.953826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a43527f-276f-4d39-bf1d-3146106459a6-tls-certs\") pod \"model-serving-api-54d89bd5fc-4jb82\" (UID: \"1a43527f-276f-4d39-bf1d-3146106459a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:26.963537 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:26.963510 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4jb82"] Jul 10 18:08:27.054815 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.054703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wh89\" (UniqueName: \"kubernetes.io/projected/1a43527f-276f-4d39-bf1d-3146106459a6-kube-api-access-7wh89\") pod \"model-serving-api-54d89bd5fc-4jb82\" (UID: \"1a43527f-276f-4d39-bf1d-3146106459a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:27.055302 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.054830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a43527f-276f-4d39-bf1d-3146106459a6-tls-certs\") pod \"model-serving-api-54d89bd5fc-4jb82\" (UID: \"1a43527f-276f-4d39-bf1d-3146106459a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:27.057546 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.057523 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a43527f-276f-4d39-bf1d-3146106459a6-tls-certs\") pod \"model-serving-api-54d89bd5fc-4jb82\" (UID: \"1a43527f-276f-4d39-bf1d-3146106459a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:27.096308 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.096273 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wh89\" (UniqueName: \"kubernetes.io/projected/1a43527f-276f-4d39-bf1d-3146106459a6-kube-api-access-7wh89\") pod \"model-serving-api-54d89bd5fc-4jb82\" (UID: \"1a43527f-276f-4d39-bf1d-3146106459a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:27.257132 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.257090 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:27.424145 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.424115 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4jb82"] Jul 10 18:08:27.426611 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:08:27.426562 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a43527f_276f_4d39_bf1d_3146106459a6.slice/crio-20ad152c6ad1bccba94dc560002c13f3fae54aa2c96ca325a80e0fc708c062c4 WatchSource:0}: Error finding container 20ad152c6ad1bccba94dc560002c13f3fae54aa2c96ca325a80e0fc708c062c4: Status 404 returned error can't find the container with id 20ad152c6ad1bccba94dc560002c13f3fae54aa2c96ca325a80e0fc708c062c4 Jul 10 18:08:27.760952 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.760902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:27.763889 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.763859 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3762dab-7ad1-440e-bbff-d8a6df59eccf-cert\") pod \"odh-model-controller-7998c67dc4-ps68r\" (UID: \"a3762dab-7ad1-440e-bbff-d8a6df59eccf\") " pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:27.774302 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.774254 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b944c"] Jul 10 18:08:27.778634 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.778584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.782401 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.782123 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 18:08:27.782401 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.782358 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 18:08:27.782583 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.782520 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 18:08:27.782676 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.782592 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bv6zc\"" Jul 10 18:08:27.783330 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.783310 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 18:08:27.792358 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.792325 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b944c"] Jul 10 18:08:27.861581 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.861546 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.861790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.861622 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.861790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.861646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-cacerts\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.861790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.861668 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-local-certs\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.861790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.861690 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-token\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.861790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.861711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.861790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.861777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjsb\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-kube-api-access-nxjsb\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.935393 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.935358 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:27.940146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.940119 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" event={"ID":"1a43527f-276f-4d39-bf1d-3146106459a6","Type":"ContainerStarted","Data":"20ad152c6ad1bccba94dc560002c13f3fae54aa2c96ca325a80e0fc708c062c4"} Jul 10 18:08:27.962659 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.962629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjsb\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-kube-api-access-nxjsb\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.962775 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.962675 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.962775 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.962707 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.962775 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.962729 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-cacerts\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.962775 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.962755 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-local-certs\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.962964 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.962860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-token\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.962964 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.962918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.963574 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.963544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.965208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.965184 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.965312 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.965230 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.965642 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.965590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-local-certs\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.965731 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.965702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-cacerts\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.993988 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.993721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjsb\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-kube-api-access-nxjsb\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:27.995466 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:27.995442 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-token\") pod \"istiod-openshift-gateway-7574459656-b944c\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:28.092278 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:28.092244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:28.130264 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:28.130190 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ps68r"] Jul 10 18:08:28.284156 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:28.283879 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b944c"] Jul 10 18:08:28.287036 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:08:28.286999 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88c0e8c_e138_4370_9d3c_8e2b50fdc106.slice/crio-49e37befa692488d5cba911c710ee4b68a4f9a6c0d61f537f29e84af1a0e48d1 WatchSource:0}: Error finding container 49e37befa692488d5cba911c710ee4b68a4f9a6c0d61f537f29e84af1a0e48d1: Status 404 returned error can't find the container with id 49e37befa692488d5cba911c710ee4b68a4f9a6c0d61f537f29e84af1a0e48d1 Jul 10 18:08:28.949618 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:28.949556 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" event={"ID":"c88c0e8c-e138-4370-9d3c-8e2b50fdc106","Type":"ContainerStarted","Data":"49e37befa692488d5cba911c710ee4b68a4f9a6c0d61f537f29e84af1a0e48d1"} Jul 10 18:08:28.950925 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:28.950888 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" event={"ID":"a3762dab-7ad1-440e-bbff-d8a6df59eccf","Type":"ContainerStarted","Data":"c6a505f6dfc53a72f5b7eb19a45a3b200e2fd08c9004a6b657dd1e926c869189"} Jul 10 18:08:32.662701 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.662652 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 18:08:32.663057 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.662743 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 18:08:32.974946 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.974908 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" event={"ID":"a3762dab-7ad1-440e-bbff-d8a6df59eccf","Type":"ContainerStarted","Data":"979d2096df4e79fbcf5d1cd4526c624728ffac1c0f659089c9e840eccdedecb4"} Jul 10 18:08:32.975130 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.975026 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:32.976876 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.976848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" event={"ID":"1a43527f-276f-4d39-bf1d-3146106459a6","Type":"ContainerStarted","Data":"49bb1f29c6f20e87b082169d29ca96336990ba9a743aa26546ac16836d0d660a"} Jul 10 18:08:32.977025 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.976919 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:32.978481 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.978457 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" event={"ID":"c88c0e8c-e138-4370-9d3c-8e2b50fdc106","Type":"ContainerStarted","Data":"20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a"} Jul 10 18:08:32.978618 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.978522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:32.980070 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.980045 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" event={"ID":"dd0ff8f4-884b-4102-bd8a-409bad35e7b6","Type":"ContainerStarted","Data":"f17ea58a0c618356fd1611f71cab0674de0c9d9c2dfa7fce059e5b0efddea8d3"} Jul 10 18:08:32.999546 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:32.999493 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" podStartSLOduration=2.471711977 podStartE2EDuration="6.999476519s" totalImagesPullingTime="4.527764542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:08:32.998040975 +0000 UTC m=+322.093231462" watchObservedRunningTime="2026-07-10 18:08:32.999476519 +0000 UTC m=+322.094667007" Jul 10 18:08:33.018888 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:33.018838 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" podStartSLOduration=1.798031291 podStartE2EDuration="7.018820174s" totalImagesPullingTime="5.220788883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:08:33.017593283 +0000 UTC m=+322.112783770" watchObservedRunningTime="2026-07-10 18:08:33.018820174 +0000 UTC m=+322.114010661" Jul 10 18:08:33.039313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:33.039259 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" podStartSLOduration=1.666976351 podStartE2EDuration="6.039245187s" totalImagesPullingTime="4.372268836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:08:33.037739524 +0000 UTC m=+322.132930011" watchObservedRunningTime="2026-07-10 18:08:33.039245187 +0000 UTC m=+322.134435673" Jul 10 18:08:33.057196 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:33.057152 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-655946c88b-vf5p4" podStartSLOduration=1.930768957 podStartE2EDuration="8.057136065s" totalImagesPullingTime="6.126367108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:08:33.055220318 +0000 UTC m=+322.150410803" watchObservedRunningTime="2026-07-10 18:08:33.057136065 +0000 UTC m=+322.152326553" Jul 10 18:08:33.986868 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:33.986839 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:37.066841 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.066805 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn"] Jul 10 18:08:37.070536 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.070515 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.073149 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.073130 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 18:08:37.073421 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.073393 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 18:08:37.074409 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.074392 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dgtkd\"" Jul 10 18:08:37.074506 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.074468 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 18:08:37.081394 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.081372 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn"] Jul 10 18:08:37.148210 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.148177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44b5858e-5bee-459d-b8cb-56495557d930-manager-config\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.148358 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.148219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44b5858e-5bee-459d-b8cb-56495557d930-metrics-cert\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.148358 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.148286 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bswq\" (UniqueName: \"kubernetes.io/projected/44b5858e-5bee-459d-b8cb-56495557d930-kube-api-access-4bswq\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.148358 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.148326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44b5858e-5bee-459d-b8cb-56495557d930-cert\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.249326 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.249292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bswq\" (UniqueName: \"kubernetes.io/projected/44b5858e-5bee-459d-b8cb-56495557d930-kube-api-access-4bswq\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.249503 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.249349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44b5858e-5bee-459d-b8cb-56495557d930-cert\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.249503 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.249380 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44b5858e-5bee-459d-b8cb-56495557d930-manager-config\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.249503 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.249418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44b5858e-5bee-459d-b8cb-56495557d930-metrics-cert\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.250184 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.250159 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44b5858e-5bee-459d-b8cb-56495557d930-manager-config\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.252105 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.252084 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44b5858e-5bee-459d-b8cb-56495557d930-cert\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.252227 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.252128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44b5858e-5bee-459d-b8cb-56495557d930-metrics-cert\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.260786 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.260757 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bswq\" (UniqueName: \"kubernetes.io/projected/44b5858e-5bee-459d-b8cb-56495557d930-kube-api-access-4bswq\") pod \"lws-controller-manager-59dbbcc5d9-qf4tn\" (UID: \"44b5858e-5bee-459d-b8cb-56495557d930\") " pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.380437 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.380341 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:37.546517 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:37.546487 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn"] Jul 10 18:08:37.548913 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:08:37.548882 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b5858e_5bee_459d_b8cb_56495557d930.slice/crio-4e7f414ff2d9908035491899906d4fed3295dd97fd376e187e7c433c1c81ce37 WatchSource:0}: Error finding container 4e7f414ff2d9908035491899906d4fed3295dd97fd376e187e7c433c1c81ce37: Status 404 returned error can't find the container with id 4e7f414ff2d9908035491899906d4fed3295dd97fd376e187e7c433c1c81ce37 Jul 10 18:08:38.002940 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:38.002900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" event={"ID":"44b5858e-5bee-459d-b8cb-56495557d930","Type":"ContainerStarted","Data":"4e7f414ff2d9908035491899906d4fed3295dd97fd376e187e7c433c1c81ce37"} Jul 10 18:08:40.013593 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.013559 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" event={"ID":"44b5858e-5bee-459d-b8cb-56495557d930","Type":"ContainerStarted","Data":"068d2f2648873edf15f2a3d1c090aa4d6e35677fccdb935e45ec26c83f5903b6"} Jul 10 18:08:40.013984 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.013709 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:08:40.050758 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.050707 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" podStartSLOduration=1.105811036 podStartE2EDuration="3.050686559s" totalImagesPullingTime="1.944875523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:08:40.048738573 +0000 UTC m=+329.143929063" watchObservedRunningTime="2026-07-10 18:08:40.050686559 +0000 UTC m=+329.145877047" Jul 10 18:08:40.153758 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.153724 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b944c"] Jul 10 18:08:40.153967 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.153938 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" podUID="c88c0e8c-e138-4370-9d3c-8e2b50fdc106" containerName="discovery" containerID="cri-o://20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a" gracePeriod=30 Jul 10 18:08:40.393042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.393017 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:40.475786 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.475753 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-cacerts\") pod \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " Jul 10 18:08:40.475959 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.475797 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-kubeconfig\") pod \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " Jul 10 18:08:40.475959 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.475833 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-local-certs\") pod \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " Jul 10 18:08:40.475959 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.475885 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-dns-cert\") pod \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " Jul 10 18:08:40.475959 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.475913 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxjsb\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-kube-api-access-nxjsb\") pod \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " Jul 10 18:08:40.475959 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.475940 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-token\") pod \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " Jul 10 18:08:40.476202 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.476030 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-ca-configmap\") pod \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\" (UID: \"c88c0e8c-e138-4370-9d3c-8e2b50fdc106\") " Jul 10 18:08:40.476592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.476548 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c88c0e8c-e138-4370-9d3c-8e2b50fdc106" (UID: "c88c0e8c-e138-4370-9d3c-8e2b50fdc106"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:08:40.478923 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.478887 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-cacerts" (OuterVolumeSpecName: "cacerts") pod "c88c0e8c-e138-4370-9d3c-8e2b50fdc106" (UID: "c88c0e8c-e138-4370-9d3c-8e2b50fdc106"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:08:40.479029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.478918 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-local-certs" (OuterVolumeSpecName: "local-certs") pod "c88c0e8c-e138-4370-9d3c-8e2b50fdc106" (UID: "c88c0e8c-e138-4370-9d3c-8e2b50fdc106"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:08:40.479029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.478968 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-kube-api-access-nxjsb" (OuterVolumeSpecName: "kube-api-access-nxjsb") pod "c88c0e8c-e138-4370-9d3c-8e2b50fdc106" (UID: "c88c0e8c-e138-4370-9d3c-8e2b50fdc106"). InnerVolumeSpecName "kube-api-access-nxjsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:08:40.479029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.478982 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c88c0e8c-e138-4370-9d3c-8e2b50fdc106" (UID: "c88c0e8c-e138-4370-9d3c-8e2b50fdc106"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:08:40.479167 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.479033 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-token" (OuterVolumeSpecName: "istio-token") pod "c88c0e8c-e138-4370-9d3c-8e2b50fdc106" (UID: "c88c0e8c-e138-4370-9d3c-8e2b50fdc106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:08:40.479167 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.479151 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c88c0e8c-e138-4370-9d3c-8e2b50fdc106" (UID: "c88c0e8c-e138-4370-9d3c-8e2b50fdc106"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:08:40.577009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.576916 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-token\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:08:40.577009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.576944 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-ca-configmap\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:08:40.577009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.576954 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-cacerts\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:08:40.577009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.576964 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-kubeconfig\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:08:40.577009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.576974 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-local-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:08:40.577009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.576982 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-istio-csr-dns-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:08:40.577009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:40.576989 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxjsb\" (UniqueName: \"kubernetes.io/projected/c88c0e8c-e138-4370-9d3c-8e2b50fdc106-kube-api-access-nxjsb\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:08:41.019331 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.019298 2570 generic.go:356] "Generic (PLEG): container finished" podID="c88c0e8c-e138-4370-9d3c-8e2b50fdc106" containerID="20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a" exitCode=0 Jul 10 18:08:41.019865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.019351 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" Jul 10 18:08:41.019865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.019389 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" event={"ID":"c88c0e8c-e138-4370-9d3c-8e2b50fdc106","Type":"ContainerDied","Data":"20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a"} Jul 10 18:08:41.019865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.019456 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b944c" event={"ID":"c88c0e8c-e138-4370-9d3c-8e2b50fdc106","Type":"ContainerDied","Data":"49e37befa692488d5cba911c710ee4b68a4f9a6c0d61f537f29e84af1a0e48d1"} Jul 10 18:08:41.019865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.019488 2570 scope.go:117] "RemoveContainer" containerID="20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a" Jul 10 18:08:41.028054 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.028035 2570 scope.go:117] "RemoveContainer" containerID="20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a" Jul 10 18:08:41.028324 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:08:41.028305 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a\": container with ID starting with 20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a not found: ID does not exist" containerID="20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a" Jul 10 18:08:41.028379 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.028333 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a"} err="failed to get container status \"20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a\": rpc error: code = NotFound desc = could not find container \"20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a\": container with ID starting with 20c93a0dbc73ebe15c1de6de0770f9e248b649e58daf3669df488c88a775c34a not found: ID does not exist" Jul 10 18:08:41.047939 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.047912 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b944c"] Jul 10 18:08:41.056910 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.056883 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b944c"] Jul 10 18:08:41.471080 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:41.471042 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88c0e8c-e138-4370-9d3c-8e2b50fdc106" path="/var/lib/kubelet/pods/c88c0e8c-e138-4370-9d3c-8e2b50fdc106/volumes" Jul 10 18:08:43.987222 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:43.987190 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ps68r" Jul 10 18:08:43.989559 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:43.989536 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4jb82" Jul 10 18:08:51.022283 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:08:51.022246 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59dbbcc5d9-qf4tn" Jul 10 18:09:42.894618 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.894566 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9"] Jul 10 18:09:42.895093 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.894883 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c0e8c-e138-4370-9d3c-8e2b50fdc106" containerName="discovery" Jul 10 18:09:42.895093 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.894894 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c0e8c-e138-4370-9d3c-8e2b50fdc106" containerName="discovery" Jul 10 18:09:42.895093 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.894951 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88c0e8c-e138-4370-9d3c-8e2b50fdc106" containerName="discovery" Jul 10 18:09:42.897136 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.897118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:42.899648 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.899628 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 18:09:42.900747 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.900722 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 18:09:42.900834 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.900764 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kdj45\"" Jul 10 18:09:42.906333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:42.906309 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9"] Jul 10 18:09:43.009824 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.009782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.009824 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.009833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfptz\" (UniqueName: \"kubernetes.io/projected/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-kube-api-access-gfptz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.010035 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.009864 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.110969 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.110932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.111158 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.111024 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.111158 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.111055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfptz\" (UniqueName: \"kubernetes.io/projected/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-kube-api-access-gfptz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.111345 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.111325 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.111383 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.111368 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.120450 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.120423 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfptz\" (UniqueName: \"kubernetes.io/projected/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-kube-api-access-gfptz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.207151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.207118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:43.344027 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.343989 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9"] Jul 10 18:09:43.352208 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:09:43.352178 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b5eebb_cae4_4902_af89_79b06ed4b7f7.slice/crio-cdeb3e38213d23f1fc5146d06cb25037eee81bf1b1742eb0b33274a61a5560ae WatchSource:0}: Error finding container cdeb3e38213d23f1fc5146d06cb25037eee81bf1b1742eb0b33274a61a5560ae: Status 404 returned error can't find the container with id cdeb3e38213d23f1fc5146d06cb25037eee81bf1b1742eb0b33274a61a5560ae Jul 10 18:09:43.494208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.494107 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk"] Jul 10 18:09:43.496960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.496934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.506329 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.506166 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk"] Jul 10 18:09:43.614729 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.614671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.614912 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.614748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6bt\" (UniqueName: \"kubernetes.io/projected/b790e7e2-11fa-4cba-9acf-a38c72615195-kube-api-access-pm6bt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.614912 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.614815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.716000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.715950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6bt\" (UniqueName: \"kubernetes.io/projected/b790e7e2-11fa-4cba-9acf-a38c72615195-kube-api-access-pm6bt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.716000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.716009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.716188 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.716056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.716398 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.716384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.716448 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.716428 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.725878 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.725843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6bt\" (UniqueName: \"kubernetes.io/projected/b790e7e2-11fa-4cba-9acf-a38c72615195-kube-api-access-pm6bt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.808669 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.808568 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:43.896307 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.896274 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9"] Jul 10 18:09:43.903117 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.903092 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:43.909301 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.909275 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9"] Jul 10 18:09:43.952313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:43.952280 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk"] Jul 10 18:09:43.957176 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:09:43.957061 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb790e7e2_11fa_4cba_9acf_a38c72615195.slice/crio-67bf57ff4bbed96d806bd520fb6cc94cb13e469563bcf0487c4e8d197ee6fa0c WatchSource:0}: Error finding container 67bf57ff4bbed96d806bd520fb6cc94cb13e469563bcf0487c4e8d197ee6fa0c: Status 404 returned error can't find the container with id 67bf57ff4bbed96d806bd520fb6cc94cb13e469563bcf0487c4e8d197ee6fa0c Jul 10 18:09:44.018328 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.018300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.018414 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.018343 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chn9\" (UniqueName: \"kubernetes.io/projected/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-kube-api-access-8chn9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.018478 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.018424 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.119608 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.119501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.119608 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.119559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.119608 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.119585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chn9\" (UniqueName: \"kubernetes.io/projected/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-kube-api-access-8chn9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.120022 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.119996 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.120096 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.120007 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.132172 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.132136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chn9\" (UniqueName: \"kubernetes.io/projected/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-kube-api-access-8chn9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.216253 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.216223 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:44.296053 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.296019 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p"] Jul 10 18:09:44.299025 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.298997 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.313251 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.313217 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p"] Jul 10 18:09:44.319211 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.319173 2570 generic.go:356] "Generic (PLEG): container finished" podID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerID="de9d5f3125b9d39d853a135088fd7b3f28210203979da8a868e3fa5fabc8f3b0" exitCode=0 Jul 10 18:09:44.319365 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.319275 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" event={"ID":"c2b5eebb-cae4-4902-af89-79b06ed4b7f7","Type":"ContainerDied","Data":"de9d5f3125b9d39d853a135088fd7b3f28210203979da8a868e3fa5fabc8f3b0"} Jul 10 18:09:44.319365 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.319348 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" event={"ID":"c2b5eebb-cae4-4902-af89-79b06ed4b7f7","Type":"ContainerStarted","Data":"cdeb3e38213d23f1fc5146d06cb25037eee81bf1b1742eb0b33274a61a5560ae"} Jul 10 18:09:44.321960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.321691 2570 generic.go:356] "Generic (PLEG): container finished" podID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerID="a7b289de967a0f7dfbf15c2aa951190d7c9e6729e9717da474987d3c362b8e80" exitCode=0 Jul 10 18:09:44.321960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.321768 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" event={"ID":"b790e7e2-11fa-4cba-9acf-a38c72615195","Type":"ContainerDied","Data":"a7b289de967a0f7dfbf15c2aa951190d7c9e6729e9717da474987d3c362b8e80"} Jul 10 18:09:44.321960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.321813 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" event={"ID":"b790e7e2-11fa-4cba-9acf-a38c72615195","Type":"ContainerStarted","Data":"67bf57ff4bbed96d806bd520fb6cc94cb13e469563bcf0487c4e8d197ee6fa0c"} Jul 10 18:09:44.362247 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.362223 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9"] Jul 10 18:09:44.364020 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:09:44.363998 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7adf02a_dabf_45ac_b0f3_35e1b5791aae.slice/crio-351d327ea8b93341643682c2b6d25e35a11cf7fabd8c448317cc159308833706 WatchSource:0}: Error finding container 351d327ea8b93341643682c2b6d25e35a11cf7fabd8c448317cc159308833706: Status 404 returned error can't find the container with id 351d327ea8b93341643682c2b6d25e35a11cf7fabd8c448317cc159308833706 Jul 10 18:09:44.422360 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.422341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.422434 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.422412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psvk\" (UniqueName: \"kubernetes.io/projected/8455b78d-7899-48ee-a4c4-f694f3b04396-kube-api-access-5psvk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.422491 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.422465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.523782 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.523750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psvk\" (UniqueName: \"kubernetes.io/projected/8455b78d-7899-48ee-a4c4-f694f3b04396-kube-api-access-5psvk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.523936 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.523808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.523936 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.523866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.524217 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.524199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.524259 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.524230 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.533167 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.533137 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psvk\" (UniqueName: \"kubernetes.io/projected/8455b78d-7899-48ee-a4c4-f694f3b04396-kube-api-access-5psvk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.610303 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.610277 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:44.759269 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:44.759241 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p"] Jul 10 18:09:44.760615 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:09:44.760573 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8455b78d_7899_48ee_a4c4_f694f3b04396.slice/crio-bf6585307fb1b19944786f9561764f9337daee7e85bfc5774f3bbd24fa935855 WatchSource:0}: Error finding container bf6585307fb1b19944786f9561764f9337daee7e85bfc5774f3bbd24fa935855: Status 404 returned error can't find the container with id bf6585307fb1b19944786f9561764f9337daee7e85bfc5774f3bbd24fa935855 Jul 10 18:09:45.328276 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.328180 2570 generic.go:356] "Generic (PLEG): container finished" podID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerID="a47fca34213efa941282cbcd1c169c5308b82ab92f0ec81afcfff67dae8f0802" exitCode=0 Jul 10 18:09:45.328762 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.328267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" event={"ID":"8455b78d-7899-48ee-a4c4-f694f3b04396","Type":"ContainerDied","Data":"a47fca34213efa941282cbcd1c169c5308b82ab92f0ec81afcfff67dae8f0802"} Jul 10 18:09:45.328762 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.328317 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" event={"ID":"8455b78d-7899-48ee-a4c4-f694f3b04396","Type":"ContainerStarted","Data":"bf6585307fb1b19944786f9561764f9337daee7e85bfc5774f3bbd24fa935855"} Jul 10 18:09:45.330553 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.330519 2570 generic.go:356] "Generic (PLEG): container finished" podID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerID="cb246978c47371498ae1214112326dcc90c3bb1048173a9172b2cc2f0f8b120e" exitCode=0 Jul 10 18:09:45.330648 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.330617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" event={"ID":"b790e7e2-11fa-4cba-9acf-a38c72615195","Type":"ContainerDied","Data":"cb246978c47371498ae1214112326dcc90c3bb1048173a9172b2cc2f0f8b120e"} Jul 10 18:09:45.332262 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.332235 2570 generic.go:356] "Generic (PLEG): container finished" podID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerID="a2496fd1afe68a141d75ab102c00ea6647c8f846c5f367d4701e7a4b21d4fdd9" exitCode=0 Jul 10 18:09:45.332370 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.332306 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" event={"ID":"a7adf02a-dabf-45ac-b0f3-35e1b5791aae","Type":"ContainerDied","Data":"a2496fd1afe68a141d75ab102c00ea6647c8f846c5f367d4701e7a4b21d4fdd9"} Jul 10 18:09:45.332480 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:45.332461 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" event={"ID":"a7adf02a-dabf-45ac-b0f3-35e1b5791aae","Type":"ContainerStarted","Data":"351d327ea8b93341643682c2b6d25e35a11cf7fabd8c448317cc159308833706"} Jul 10 18:09:46.339357 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.339314 2570 generic.go:356] "Generic (PLEG): container finished" podID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerID="85a0a84bf04c7aa78996a135938d859bffd842c549fe873b9f3297e305099e18" exitCode=0 Jul 10 18:09:46.339807 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.339405 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" event={"ID":"b790e7e2-11fa-4cba-9acf-a38c72615195","Type":"ContainerDied","Data":"85a0a84bf04c7aa78996a135938d859bffd842c549fe873b9f3297e305099e18"} Jul 10 18:09:46.341287 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.341260 2570 generic.go:356] "Generic (PLEG): container finished" podID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerID="f2b0077896a31306ae336d50922d9c8aeb5b4101bf733e6b3dfcf65e60a01b45" exitCode=0 Jul 10 18:09:46.341403 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.341380 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" event={"ID":"a7adf02a-dabf-45ac-b0f3-35e1b5791aae","Type":"ContainerDied","Data":"f2b0077896a31306ae336d50922d9c8aeb5b4101bf733e6b3dfcf65e60a01b45"} Jul 10 18:09:46.343393 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.343368 2570 generic.go:356] "Generic (PLEG): container finished" podID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerID="d88e6788f63574fc5c6bb01d49518be48e283e61ca1fdff6bd9caa2175515bb9" exitCode=0 Jul 10 18:09:46.343480 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.343434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" event={"ID":"8455b78d-7899-48ee-a4c4-f694f3b04396","Type":"ContainerDied","Data":"d88e6788f63574fc5c6bb01d49518be48e283e61ca1fdff6bd9caa2175515bb9"} Jul 10 18:09:46.345345 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.345326 2570 generic.go:356] "Generic (PLEG): container finished" podID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerID="5c36df4d4c239600f29e621e8cd90276cb028d1d44abbee913f48a36d868584c" exitCode=0 Jul 10 18:09:46.345452 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:46.345403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" event={"ID":"c2b5eebb-cae4-4902-af89-79b06ed4b7f7","Type":"ContainerDied","Data":"5c36df4d4c239600f29e621e8cd90276cb028d1d44abbee913f48a36d868584c"} Jul 10 18:09:47.352608 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.352559 2570 generic.go:356] "Generic (PLEG): container finished" podID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerID="a455259f23deda9bed4cf584e741bce6928f5786f1fa8a7bc0a5330e3f98458a" exitCode=0 Jul 10 18:09:47.353013 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.352647 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" event={"ID":"8455b78d-7899-48ee-a4c4-f694f3b04396","Type":"ContainerDied","Data":"a455259f23deda9bed4cf584e741bce6928f5786f1fa8a7bc0a5330e3f98458a"} Jul 10 18:09:47.354616 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.354575 2570 generic.go:356] "Generic (PLEG): container finished" podID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerID="35f1eb18b643b4ea34e12b5eb6c920ea90f9e430b8688e42a438ac4cb8eee04b" exitCode=0 Jul 10 18:09:47.354714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.354695 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" event={"ID":"c2b5eebb-cae4-4902-af89-79b06ed4b7f7","Type":"ContainerDied","Data":"35f1eb18b643b4ea34e12b5eb6c920ea90f9e430b8688e42a438ac4cb8eee04b"} Jul 10 18:09:47.356564 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.356541 2570 generic.go:356] "Generic (PLEG): container finished" podID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerID="9df1b36ec34c8882f085384798315384672f396748b9ca30fcef9ef67259c0d5" exitCode=0 Jul 10 18:09:47.356733 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.356569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" event={"ID":"a7adf02a-dabf-45ac-b0f3-35e1b5791aae","Type":"ContainerDied","Data":"9df1b36ec34c8882f085384798315384672f396748b9ca30fcef9ef67259c0d5"} Jul 10 18:09:47.487882 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.487855 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:47.552222 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.552186 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm6bt\" (UniqueName: \"kubernetes.io/projected/b790e7e2-11fa-4cba-9acf-a38c72615195-kube-api-access-pm6bt\") pod \"b790e7e2-11fa-4cba-9acf-a38c72615195\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " Jul 10 18:09:47.552390 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.552248 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-bundle\") pod \"b790e7e2-11fa-4cba-9acf-a38c72615195\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " Jul 10 18:09:47.552390 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.552346 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-util\") pod \"b790e7e2-11fa-4cba-9acf-a38c72615195\" (UID: \"b790e7e2-11fa-4cba-9acf-a38c72615195\") " Jul 10 18:09:47.553050 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.553024 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-bundle" (OuterVolumeSpecName: "bundle") pod "b790e7e2-11fa-4cba-9acf-a38c72615195" (UID: "b790e7e2-11fa-4cba-9acf-a38c72615195"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:47.554626 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.554590 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b790e7e2-11fa-4cba-9acf-a38c72615195-kube-api-access-pm6bt" (OuterVolumeSpecName: "kube-api-access-pm6bt") pod "b790e7e2-11fa-4cba-9acf-a38c72615195" (UID: "b790e7e2-11fa-4cba-9acf-a38c72615195"). InnerVolumeSpecName "kube-api-access-pm6bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:09:47.557802 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.557782 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-util" (OuterVolumeSpecName: "util") pod "b790e7e2-11fa-4cba-9acf-a38c72615195" (UID: "b790e7e2-11fa-4cba-9acf-a38c72615195"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:47.653696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.653662 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-util\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.653696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.653691 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6bt\" (UniqueName: \"kubernetes.io/projected/b790e7e2-11fa-4cba-9acf-a38c72615195-kube-api-access-pm6bt\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.653696 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:47.653700 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b790e7e2-11fa-4cba-9acf-a38c72615195-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.362756 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.362730 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" Jul 10 18:09:48.362756 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.362733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk" event={"ID":"b790e7e2-11fa-4cba-9acf-a38c72615195","Type":"ContainerDied","Data":"67bf57ff4bbed96d806bd520fb6cc94cb13e469563bcf0487c4e8d197ee6fa0c"} Jul 10 18:09:48.363181 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.362785 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67bf57ff4bbed96d806bd520fb6cc94cb13e469563bcf0487c4e8d197ee6fa0c" Jul 10 18:09:48.497457 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.497423 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:48.538421 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.538389 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:48.541537 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.541516 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:48.562591 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.562570 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-util\") pod \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " Jul 10 18:09:48.562711 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.562646 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-bundle\") pod \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " Jul 10 18:09:48.562711 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.562686 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfptz\" (UniqueName: \"kubernetes.io/projected/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-kube-api-access-gfptz\") pod \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\" (UID: \"c2b5eebb-cae4-4902-af89-79b06ed4b7f7\") " Jul 10 18:09:48.563205 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.563175 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-bundle" (OuterVolumeSpecName: "bundle") pod "c2b5eebb-cae4-4902-af89-79b06ed4b7f7" (UID: "c2b5eebb-cae4-4902-af89-79b06ed4b7f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:48.565051 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.565019 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-kube-api-access-gfptz" (OuterVolumeSpecName: "kube-api-access-gfptz") pod "c2b5eebb-cae4-4902-af89-79b06ed4b7f7" (UID: "c2b5eebb-cae4-4902-af89-79b06ed4b7f7"). InnerVolumeSpecName "kube-api-access-gfptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:09:48.568879 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.568851 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-util" (OuterVolumeSpecName: "util") pod "c2b5eebb-cae4-4902-af89-79b06ed4b7f7" (UID: "c2b5eebb-cae4-4902-af89-79b06ed4b7f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:48.663674 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.663644 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5psvk\" (UniqueName: \"kubernetes.io/projected/8455b78d-7899-48ee-a4c4-f694f3b04396-kube-api-access-5psvk\") pod \"8455b78d-7899-48ee-a4c4-f694f3b04396\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " Jul 10 18:09:48.663830 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.663713 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-util\") pod \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " Jul 10 18:09:48.663830 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.663740 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-bundle\") pod \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " Jul 10 18:09:48.663830 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.663765 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-bundle\") pod \"8455b78d-7899-48ee-a4c4-f694f3b04396\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " Jul 10 18:09:48.663830 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.663793 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8chn9\" (UniqueName: \"kubernetes.io/projected/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-kube-api-access-8chn9\") pod \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\" (UID: \"a7adf02a-dabf-45ac-b0f3-35e1b5791aae\") " Jul 10 18:09:48.663830 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.663809 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-util\") pod \"8455b78d-7899-48ee-a4c4-f694f3b04396\" (UID: \"8455b78d-7899-48ee-a4c4-f694f3b04396\") " Jul 10 18:09:48.664087 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.663993 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-util\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.664087 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.664009 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.664087 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.664023 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfptz\" (UniqueName: \"kubernetes.io/projected/c2b5eebb-cae4-4902-af89-79b06ed4b7f7-kube-api-access-gfptz\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.664336 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.664260 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-bundle" (OuterVolumeSpecName: "bundle") pod "a7adf02a-dabf-45ac-b0f3-35e1b5791aae" (UID: "a7adf02a-dabf-45ac-b0f3-35e1b5791aae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:48.664336 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.664274 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-bundle" (OuterVolumeSpecName: "bundle") pod "8455b78d-7899-48ee-a4c4-f694f3b04396" (UID: "8455b78d-7899-48ee-a4c4-f694f3b04396"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:48.666052 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.666030 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8455b78d-7899-48ee-a4c4-f694f3b04396-kube-api-access-5psvk" (OuterVolumeSpecName: "kube-api-access-5psvk") pod "8455b78d-7899-48ee-a4c4-f694f3b04396" (UID: "8455b78d-7899-48ee-a4c4-f694f3b04396"). InnerVolumeSpecName "kube-api-access-5psvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:09:48.666360 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.666334 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-kube-api-access-8chn9" (OuterVolumeSpecName: "kube-api-access-8chn9") pod "a7adf02a-dabf-45ac-b0f3-35e1b5791aae" (UID: "a7adf02a-dabf-45ac-b0f3-35e1b5791aae"). InnerVolumeSpecName "kube-api-access-8chn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:09:48.668818 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.668795 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-util" (OuterVolumeSpecName: "util") pod "a7adf02a-dabf-45ac-b0f3-35e1b5791aae" (UID: "a7adf02a-dabf-45ac-b0f3-35e1b5791aae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:48.670410 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.670388 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-util" (OuterVolumeSpecName: "util") pod "8455b78d-7899-48ee-a4c4-f694f3b04396" (UID: "8455b78d-7899-48ee-a4c4-f694f3b04396"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:48.765288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.765256 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psvk\" (UniqueName: \"kubernetes.io/projected/8455b78d-7899-48ee-a4c4-f694f3b04396-kube-api-access-5psvk\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.765288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.765283 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-util\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.765288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.765296 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.765492 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.765305 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.765492 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.765313 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chn9\" (UniqueName: \"kubernetes.io/projected/a7adf02a-dabf-45ac-b0f3-35e1b5791aae-kube-api-access-8chn9\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:48.765492 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:48.765321 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8455b78d-7899-48ee-a4c4-f694f3b04396-util\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:09:49.369721 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.369694 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" Jul 10 18:09:49.369721 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.369706 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p" event={"ID":"8455b78d-7899-48ee-a4c4-f694f3b04396","Type":"ContainerDied","Data":"bf6585307fb1b19944786f9561764f9337daee7e85bfc5774f3bbd24fa935855"} Jul 10 18:09:49.370170 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.369758 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf6585307fb1b19944786f9561764f9337daee7e85bfc5774f3bbd24fa935855" Jul 10 18:09:49.371715 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.371682 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" Jul 10 18:09:49.371715 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.371689 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9" event={"ID":"c2b5eebb-cae4-4902-af89-79b06ed4b7f7","Type":"ContainerDied","Data":"cdeb3e38213d23f1fc5146d06cb25037eee81bf1b1742eb0b33274a61a5560ae"} Jul 10 18:09:49.371890 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.371732 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdeb3e38213d23f1fc5146d06cb25037eee81bf1b1742eb0b33274a61a5560ae" Jul 10 18:09:49.381303 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.381269 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" event={"ID":"a7adf02a-dabf-45ac-b0f3-35e1b5791aae","Type":"ContainerDied","Data":"351d327ea8b93341643682c2b6d25e35a11cf7fabd8c448317cc159308833706"} Jul 10 18:09:49.381473 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.381357 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351d327ea8b93341643682c2b6d25e35a11cf7fabd8c448317cc159308833706" Jul 10 18:09:49.381473 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:49.381284 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9" Jul 10 18:09:55.484320 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:09:55.484284 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdfb5b968-hbph7"] Jul 10 18:10:09.725429 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725385 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r"] Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725713 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerName="pull" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725726 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerName="pull" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725738 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerName="pull" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725743 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerName="pull" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725750 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725756 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725770 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725775 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725782 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725787 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725794 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725799 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725805 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerName="pull" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725809 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerName="pull" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725815 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725820 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725828 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725835 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725840 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725845 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerName="util" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725850 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725857 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerName="extract" Jul 10 18:10:09.725845 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725863 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerName="pull" Jul 10 18:10:09.726586 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725868 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerName="pull" Jul 10 18:10:09.726586 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725912 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b5eebb-cae4-4902-af89-79b06ed4b7f7" containerName="extract" Jul 10 18:10:09.726586 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725919 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8455b78d-7899-48ee-a4c4-f694f3b04396" containerName="extract" Jul 10 18:10:09.726586 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725927 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7adf02a-dabf-45ac-b0f3-35e1b5791aae" containerName="extract" Jul 10 18:10:09.726586 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.725933 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b790e7e2-11fa-4cba-9acf-a38c72615195" containerName="extract" Jul 10 18:10:09.728077 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.728058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.730795 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.730758 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kdj45\"" Jul 10 18:10:09.730914 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.730806 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 18:10:09.730914 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.730820 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 18:10:09.731859 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.731845 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 18:10:09.731916 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.731874 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 18:10:09.740717 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.740695 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r"] Jul 10 18:10:09.852266 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.852229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36aab6cf-13c3-4a26-9127-de4e8f67cd31-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.852414 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.852275 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcnh\" (UniqueName: \"kubernetes.io/projected/36aab6cf-13c3-4a26-9127-de4e8f67cd31-kube-api-access-bjcnh\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.852414 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.852309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/36aab6cf-13c3-4a26-9127-de4e8f67cd31-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.953048 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.953012 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcnh\" (UniqueName: \"kubernetes.io/projected/36aab6cf-13c3-4a26-9127-de4e8f67cd31-kube-api-access-bjcnh\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.953048 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.953056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/36aab6cf-13c3-4a26-9127-de4e8f67cd31-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.953279 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.953126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36aab6cf-13c3-4a26-9127-de4e8f67cd31-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.953960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.953891 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36aab6cf-13c3-4a26-9127-de4e8f67cd31-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.955795 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.955776 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/36aab6cf-13c3-4a26-9127-de4e8f67cd31-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:09.970986 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:09.970961 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcnh\" (UniqueName: \"kubernetes.io/projected/36aab6cf-13c3-4a26-9127-de4e8f67cd31-kube-api-access-bjcnh\") pod \"kuadrant-console-plugin-66b5774985-x7z2r\" (UID: \"36aab6cf-13c3-4a26-9127-de4e8f67cd31\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:10.037678 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:10.037570 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" Jul 10 18:10:10.190540 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:10.190510 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r"] Jul 10 18:10:10.192509 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:10:10.192482 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36aab6cf_13c3_4a26_9127_de4e8f67cd31.slice/crio-91ea6f8ef0759d440b76847c86202b93a2dcd36d8a40f47511559179fa8e14b9 WatchSource:0}: Error finding container 91ea6f8ef0759d440b76847c86202b93a2dcd36d8a40f47511559179fa8e14b9: Status 404 returned error can't find the container with id 91ea6f8ef0759d440b76847c86202b93a2dcd36d8a40f47511559179fa8e14b9 Jul 10 18:10:10.485175 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:10.485137 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" event={"ID":"36aab6cf-13c3-4a26-9127-de4e8f67cd31","Type":"ContainerStarted","Data":"91ea6f8ef0759d440b76847c86202b93a2dcd36d8a40f47511559179fa8e14b9"} Jul 10 18:10:20.505208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:20.505156 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cdfb5b968-hbph7" podUID="7672036c-c69a-44e1-a8bf-04512ba8213a" containerName="console" containerID="cri-o://f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199" gracePeriod=15 Jul 10 18:10:24.342673 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:24.342634 2570 patch_prober.go:28] interesting pod/console-7cdfb5b968-hbph7 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body= Jul 10 18:10:24.343111 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:24.342705 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7cdfb5b968-hbph7" podUID="7672036c-c69a-44e1-a8bf-04512ba8213a" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" Jul 10 18:10:34.343084 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.342989 2570 patch_prober.go:28] interesting pod/console-7cdfb5b968-hbph7 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body= Jul 10 18:10:34.343084 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.343057 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7cdfb5b968-hbph7" podUID="7672036c-c69a-44e1-a8bf-04512ba8213a" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" Jul 10 18:10:34.582308 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.582282 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:10:34.624425 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.624345 2570 generic.go:356] "Generic (PLEG): container finished" podID="7672036c-c69a-44e1-a8bf-04512ba8213a" containerID="f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199" exitCode=0 Jul 10 18:10:34.624425 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.624393 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdfb5b968-hbph7" Jul 10 18:10:34.624425 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.624411 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdfb5b968-hbph7" event={"ID":"7672036c-c69a-44e1-a8bf-04512ba8213a","Type":"ContainerDied","Data":"f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199"} Jul 10 18:10:34.624694 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.624460 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdfb5b968-hbph7" event={"ID":"7672036c-c69a-44e1-a8bf-04512ba8213a","Type":"ContainerDied","Data":"6d009fe872034e45b368329dd12031b26fba68e84ea67aa8392534a9899326e3"} Jul 10 18:10:34.624694 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.624481 2570 scope.go:117] "RemoveContainer" containerID="f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199" Jul 10 18:10:34.626149 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.626126 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" event={"ID":"36aab6cf-13c3-4a26-9127-de4e8f67cd31","Type":"ContainerStarted","Data":"237e95f3c996e70b88f1ebbb54a5b36d192053f1622050339662e101efc8a1a1"} Jul 10 18:10:34.634683 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.634666 2570 scope.go:117] "RemoveContainer" containerID="f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199" Jul 10 18:10:34.634960 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:10:34.634942 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199\": container with ID starting with f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199 not found: ID does not exist" containerID="f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199" Jul 10 18:10:34.635014 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.634967 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199"} err="failed to get container status \"f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199\": rpc error: code = NotFound desc = could not find container \"f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199\": container with ID starting with f35bfaa92e226e7dd20312dc7fb26ccca1d27d6b7a280ddb72af8853f7801199 not found: ID does not exist" Jul 10 18:10:34.651126 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.651082 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x7z2r" podStartSLOduration=1.3387712139999999 podStartE2EDuration="25.651068814s" totalImagesPullingTime="24.3122976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:10:34.649655258 +0000 UTC m=+443.744845735" watchObservedRunningTime="2026-07-10 18:10:34.651068814 +0000 UTC m=+443.746259300" Jul 10 18:10:34.686094 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686071 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-console-config\") pod \"7672036c-c69a-44e1-a8bf-04512ba8213a\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " Jul 10 18:10:34.686204 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686136 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-serving-cert\") pod \"7672036c-c69a-44e1-a8bf-04512ba8213a\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " Jul 10 18:10:34.686204 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686156 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-oauth-serving-cert\") pod \"7672036c-c69a-44e1-a8bf-04512ba8213a\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " Jul 10 18:10:34.686204 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686172 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-oauth-config\") pod \"7672036c-c69a-44e1-a8bf-04512ba8213a\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " Jul 10 18:10:34.686353 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686205 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-service-ca\") pod \"7672036c-c69a-44e1-a8bf-04512ba8213a\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " Jul 10 18:10:34.686353 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686335 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-trusted-ca-bundle\") pod \"7672036c-c69a-44e1-a8bf-04512ba8213a\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " Jul 10 18:10:34.686450 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686422 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqhnd\" (UniqueName: \"kubernetes.io/projected/7672036c-c69a-44e1-a8bf-04512ba8213a-kube-api-access-fqhnd\") pod \"7672036c-c69a-44e1-a8bf-04512ba8213a\" (UID: \"7672036c-c69a-44e1-a8bf-04512ba8213a\") " Jul 10 18:10:34.686731 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686522 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-console-config" (OuterVolumeSpecName: "console-config") pod "7672036c-c69a-44e1-a8bf-04512ba8213a" (UID: "7672036c-c69a-44e1-a8bf-04512ba8213a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:10:34.686851 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686665 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7672036c-c69a-44e1-a8bf-04512ba8213a" (UID: "7672036c-c69a-44e1-a8bf-04512ba8213a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:10:34.686851 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686618 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7672036c-c69a-44e1-a8bf-04512ba8213a" (UID: "7672036c-c69a-44e1-a8bf-04512ba8213a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:10:34.686995 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.686973 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-oauth-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:10:34.687083 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.687001 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-service-ca\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:10:34.687083 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.687017 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-console-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:10:34.687083 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.687035 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7672036c-c69a-44e1-a8bf-04512ba8213a" (UID: "7672036c-c69a-44e1-a8bf-04512ba8213a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:10:34.688736 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.688711 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7672036c-c69a-44e1-a8bf-04512ba8213a-kube-api-access-fqhnd" (OuterVolumeSpecName: "kube-api-access-fqhnd") pod "7672036c-c69a-44e1-a8bf-04512ba8213a" (UID: "7672036c-c69a-44e1-a8bf-04512ba8213a"). InnerVolumeSpecName "kube-api-access-fqhnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:10:34.688922 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.688904 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7672036c-c69a-44e1-a8bf-04512ba8213a" (UID: "7672036c-c69a-44e1-a8bf-04512ba8213a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:10:34.688989 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.688927 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7672036c-c69a-44e1-a8bf-04512ba8213a" (UID: "7672036c-c69a-44e1-a8bf-04512ba8213a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:10:34.787499 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.787456 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-serving-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:10:34.787499 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.787494 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7672036c-c69a-44e1-a8bf-04512ba8213a-console-oauth-config\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:10:34.787499 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.787506 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7672036c-c69a-44e1-a8bf-04512ba8213a-trusted-ca-bundle\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:10:34.787764 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.787516 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqhnd\" (UniqueName: \"kubernetes.io/projected/7672036c-c69a-44e1-a8bf-04512ba8213a-kube-api-access-fqhnd\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:10:34.949078 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.949046 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdfb5b968-hbph7"] Jul 10 18:10:34.953862 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:34.953838 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cdfb5b968-hbph7"] Jul 10 18:10:35.463877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:10:35.463842 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7672036c-c69a-44e1-a8bf-04512ba8213a" path="/var/lib/kubelet/pods/7672036c-c69a-44e1-a8bf-04512ba8213a/volumes" Jul 10 18:11:01.791651 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.791619 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-vvcv8"] Jul 10 18:11:01.792111 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.791932 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7672036c-c69a-44e1-a8bf-04512ba8213a" containerName="console" Jul 10 18:11:01.792111 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.791945 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7672036c-c69a-44e1-a8bf-04512ba8213a" containerName="console" Jul 10 18:11:01.792111 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.792014 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7672036c-c69a-44e1-a8bf-04512ba8213a" containerName="console" Jul 10 18:11:01.961359 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.961330 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-vvcv8"] Jul 10 18:11:01.961359 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.961364 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lnjk6"] Jul 10 18:11:01.961573 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.961455 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vvcv8" Jul 10 18:11:01.964416 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.964394 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pdwm2\"" Jul 10 18:11:01.972019 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.972001 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lnjk6"] Jul 10 18:11:01.972114 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:01.972073 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lnjk6" Jul 10 18:11:02.009691 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.009658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bsv\" (UniqueName: \"kubernetes.io/projected/8853f240-a7b0-462c-a2f0-6f5e3fcd351b-kube-api-access-45bsv\") pod \"authorino-58cd955675-vvcv8\" (UID: \"8853f240-a7b0-462c-a2f0-6f5e3fcd351b\") " pod="kuadrant-system/authorino-58cd955675-vvcv8" Jul 10 18:11:02.009846 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.009712 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckkq\" (UniqueName: \"kubernetes.io/projected/e2875c77-3782-4653-af42-f0c089216740-kube-api-access-zckkq\") pod \"authorino-75d556bd77-lnjk6\" (UID: \"e2875c77-3782-4653-af42-f0c089216740\") " pod="kuadrant-system/authorino-75d556bd77-lnjk6" Jul 10 18:11:02.110584 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.110490 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bsv\" (UniqueName: \"kubernetes.io/projected/8853f240-a7b0-462c-a2f0-6f5e3fcd351b-kube-api-access-45bsv\") pod \"authorino-58cd955675-vvcv8\" (UID: \"8853f240-a7b0-462c-a2f0-6f5e3fcd351b\") " pod="kuadrant-system/authorino-58cd955675-vvcv8" Jul 10 18:11:02.110584 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.110553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckkq\" (UniqueName: \"kubernetes.io/projected/e2875c77-3782-4653-af42-f0c089216740-kube-api-access-zckkq\") pod \"authorino-75d556bd77-lnjk6\" (UID: \"e2875c77-3782-4653-af42-f0c089216740\") " pod="kuadrant-system/authorino-75d556bd77-lnjk6" Jul 10 18:11:02.123182 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.123153 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckkq\" (UniqueName: \"kubernetes.io/projected/e2875c77-3782-4653-af42-f0c089216740-kube-api-access-zckkq\") pod \"authorino-75d556bd77-lnjk6\" (UID: \"e2875c77-3782-4653-af42-f0c089216740\") " pod="kuadrant-system/authorino-75d556bd77-lnjk6" Jul 10 18:11:02.127331 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.127312 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bsv\" (UniqueName: \"kubernetes.io/projected/8853f240-a7b0-462c-a2f0-6f5e3fcd351b-kube-api-access-45bsv\") pod \"authorino-58cd955675-vvcv8\" (UID: \"8853f240-a7b0-462c-a2f0-6f5e3fcd351b\") " pod="kuadrant-system/authorino-58cd955675-vvcv8" Jul 10 18:11:02.270706 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.270660 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vvcv8" Jul 10 18:11:02.280840 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.280808 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lnjk6" Jul 10 18:11:02.518476 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.518442 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-vvcv8"] Jul 10 18:11:02.521398 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:11:02.521372 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8853f240_a7b0_462c_a2f0_6f5e3fcd351b.slice/crio-f3687cd35ccd7099a3e07ada70a1be8663c448f508ad7c0ed00e0825c8e3c09e WatchSource:0}: Error finding container f3687cd35ccd7099a3e07ada70a1be8663c448f508ad7c0ed00e0825c8e3c09e: Status 404 returned error can't find the container with id f3687cd35ccd7099a3e07ada70a1be8663c448f508ad7c0ed00e0825c8e3c09e Jul 10 18:11:02.544566 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.544543 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lnjk6"] Jul 10 18:11:02.546570 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:11:02.546539 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2875c77_3782_4653_af42_f0c089216740.slice/crio-c58cc8309475eaa1abfce840c22b699ccac6b4a31175ddbecdcc9bf17f7310f9 WatchSource:0}: Error finding container c58cc8309475eaa1abfce840c22b699ccac6b4a31175ddbecdcc9bf17f7310f9: Status 404 returned error can't find the container with id c58cc8309475eaa1abfce840c22b699ccac6b4a31175ddbecdcc9bf17f7310f9 Jul 10 18:11:02.760866 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.760826 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lnjk6" event={"ID":"e2875c77-3782-4653-af42-f0c089216740","Type":"ContainerStarted","Data":"c58cc8309475eaa1abfce840c22b699ccac6b4a31175ddbecdcc9bf17f7310f9"} Jul 10 18:11:02.761950 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:02.761926 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vvcv8" event={"ID":"8853f240-a7b0-462c-a2f0-6f5e3fcd351b","Type":"ContainerStarted","Data":"f3687cd35ccd7099a3e07ada70a1be8663c448f508ad7c0ed00e0825c8e3c09e"} Jul 10 18:11:06.790668 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:06.790627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vvcv8" event={"ID":"8853f240-a7b0-462c-a2f0-6f5e3fcd351b","Type":"ContainerStarted","Data":"400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2"} Jul 10 18:11:06.792089 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:06.792059 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lnjk6" event={"ID":"e2875c77-3782-4653-af42-f0c089216740","Type":"ContainerStarted","Data":"33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77"} Jul 10 18:11:06.811191 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:06.811131 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-vvcv8" podStartSLOduration=1.959770077 podStartE2EDuration="5.811114812s" totalImagesPullingTime="3.851344735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:11:06.810249144 +0000 UTC m=+475.905439631" watchObservedRunningTime="2026-07-10 18:11:06.811114812 +0000 UTC m=+475.906305330" Jul 10 18:11:06.868208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:06.868149 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lnjk6" podStartSLOduration=2.030435847 podStartE2EDuration="5.868132446s" totalImagesPullingTime="3.837696599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:11:06.838624352 +0000 UTC m=+475.933814829" watchObservedRunningTime="2026-07-10 18:11:06.868132446 +0000 UTC m=+475.963322931" Jul 10 18:11:06.869079 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:06.869062 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-vvcv8"] Jul 10 18:11:08.801076 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:08.801030 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-vvcv8" podUID="8853f240-a7b0-462c-a2f0-6f5e3fcd351b" containerName="authorino" containerID="cri-o://400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2" gracePeriod=30 Jul 10 18:11:09.051258 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.051206 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vvcv8" Jul 10 18:11:09.171430 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.171382 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45bsv\" (UniqueName: \"kubernetes.io/projected/8853f240-a7b0-462c-a2f0-6f5e3fcd351b-kube-api-access-45bsv\") pod \"8853f240-a7b0-462c-a2f0-6f5e3fcd351b\" (UID: \"8853f240-a7b0-462c-a2f0-6f5e3fcd351b\") " Jul 10 18:11:09.173593 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.173572 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8853f240-a7b0-462c-a2f0-6f5e3fcd351b-kube-api-access-45bsv" (OuterVolumeSpecName: "kube-api-access-45bsv") pod "8853f240-a7b0-462c-a2f0-6f5e3fcd351b" (UID: "8853f240-a7b0-462c-a2f0-6f5e3fcd351b"). InnerVolumeSpecName "kube-api-access-45bsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:09.272469 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.272440 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45bsv\" (UniqueName: \"kubernetes.io/projected/8853f240-a7b0-462c-a2f0-6f5e3fcd351b-kube-api-access-45bsv\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:11:09.807166 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.807076 2570 generic.go:356] "Generic (PLEG): container finished" podID="8853f240-a7b0-462c-a2f0-6f5e3fcd351b" containerID="400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2" exitCode=0 Jul 10 18:11:09.807166 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.807122 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vvcv8" Jul 10 18:11:09.807657 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.807167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vvcv8" event={"ID":"8853f240-a7b0-462c-a2f0-6f5e3fcd351b","Type":"ContainerDied","Data":"400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2"} Jul 10 18:11:09.807657 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.807213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vvcv8" event={"ID":"8853f240-a7b0-462c-a2f0-6f5e3fcd351b","Type":"ContainerDied","Data":"f3687cd35ccd7099a3e07ada70a1be8663c448f508ad7c0ed00e0825c8e3c09e"} Jul 10 18:11:09.807657 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.807230 2570 scope.go:117] "RemoveContainer" containerID="400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2" Jul 10 18:11:09.815695 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.815679 2570 scope.go:117] "RemoveContainer" containerID="400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2" Jul 10 18:11:09.815941 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:11:09.815920 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2\": container with ID starting with 400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2 not found: ID does not exist" containerID="400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2" Jul 10 18:11:09.816008 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.815948 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2"} err="failed to get container status \"400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2\": rpc error: code = NotFound desc = could not find container \"400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2\": container with ID starting with 400228625c3193fe43d0d215fda80b46ccc5321e0f8fc562df0933c1f1b1c1a2 not found: ID does not exist" Jul 10 18:11:09.827588 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.827560 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-vvcv8"] Jul 10 18:11:09.832181 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:09.832157 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-vvcv8"] Jul 10 18:11:11.464335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:11.464301 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8853f240-a7b0-462c-a2f0-6f5e3fcd351b" path="/var/lib/kubelet/pods/8853f240-a7b0-462c-a2f0-6f5e3fcd351b/volumes" Jul 10 18:11:29.521775 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.521737 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ss2lx"] Jul 10 18:11:29.522341 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.522157 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8853f240-a7b0-462c-a2f0-6f5e3fcd351b" containerName="authorino" Jul 10 18:11:29.522341 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.522175 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8853f240-a7b0-462c-a2f0-6f5e3fcd351b" containerName="authorino" Jul 10 18:11:29.522341 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.522252 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8853f240-a7b0-462c-a2f0-6f5e3fcd351b" containerName="authorino" Jul 10 18:11:29.526552 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.526531 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" Jul 10 18:11:29.536525 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.536501 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ss2lx"] Jul 10 18:11:29.640727 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.640702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpdn\" (UniqueName: \"kubernetes.io/projected/c9b4defc-5343-456d-82ba-bc16c7eee1cf-kube-api-access-kdpdn\") pod \"authorino-5d4b67c446-ss2lx\" (UID: \"c9b4defc-5343-456d-82ba-bc16c7eee1cf\") " pod="kuadrant-system/authorino-5d4b67c446-ss2lx" Jul 10 18:11:29.741442 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.741416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpdn\" (UniqueName: \"kubernetes.io/projected/c9b4defc-5343-456d-82ba-bc16c7eee1cf-kube-api-access-kdpdn\") pod \"authorino-5d4b67c446-ss2lx\" (UID: \"c9b4defc-5343-456d-82ba-bc16c7eee1cf\") " pod="kuadrant-system/authorino-5d4b67c446-ss2lx" Jul 10 18:11:29.755118 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.755090 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpdn\" (UniqueName: \"kubernetes.io/projected/c9b4defc-5343-456d-82ba-bc16c7eee1cf-kube-api-access-kdpdn\") pod \"authorino-5d4b67c446-ss2lx\" (UID: \"c9b4defc-5343-456d-82ba-bc16c7eee1cf\") " pod="kuadrant-system/authorino-5d4b67c446-ss2lx" Jul 10 18:11:29.768639 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.768615 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ss2lx"] Jul 10 18:11:29.768819 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.768807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" Jul 10 18:11:29.797136 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.797102 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-749659456f-j8f2s"] Jul 10 18:11:29.801960 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.801937 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749659456f-j8f2s" Jul 10 18:11:29.818468 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.818427 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-749659456f-j8f2s"] Jul 10 18:11:29.910608 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.910564 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ss2lx"] Jul 10 18:11:29.913575 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:11:29.913545 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9b4defc_5343_456d_82ba_bc16c7eee1cf.slice/crio-8645aa3f78e44ec284c7d134f46fed99796b073d94d56019f25d9f47e7eced6f WatchSource:0}: Error finding container 8645aa3f78e44ec284c7d134f46fed99796b073d94d56019f25d9f47e7eced6f: Status 404 returned error can't find the container with id 8645aa3f78e44ec284c7d134f46fed99796b073d94d56019f25d9f47e7eced6f Jul 10 18:11:29.938656 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.938631 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-749659456f-j8f2s"] Jul 10 18:11:29.938869 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:11:29.938853 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qrlmn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-749659456f-j8f2s" podUID="c34bfb3d-06b9-4ff8-ab81-d44577f84b27" Jul 10 18:11:29.943267 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.943244 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlmn\" (UniqueName: \"kubernetes.io/projected/c34bfb3d-06b9-4ff8-ab81-d44577f84b27-kube-api-access-qrlmn\") pod \"authorino-749659456f-j8f2s\" (UID: \"c34bfb3d-06b9-4ff8-ab81-d44577f84b27\") " pod="kuadrant-system/authorino-749659456f-j8f2s" Jul 10 18:11:29.974224 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.974196 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c45d58d97-rxdfp"] Jul 10 18:11:29.977810 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.977795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:29.981164 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:29.981144 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 18:11:30.000139 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.000117 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c45d58d97-rxdfp"] Jul 10 18:11:30.044227 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.044159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlmn\" (UniqueName: \"kubernetes.io/projected/c34bfb3d-06b9-4ff8-ab81-d44577f84b27-kube-api-access-qrlmn\") pod \"authorino-749659456f-j8f2s\" (UID: \"c34bfb3d-06b9-4ff8-ab81-d44577f84b27\") " pod="kuadrant-system/authorino-749659456f-j8f2s" Jul 10 18:11:30.044338 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.044222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfwq\" (UniqueName: \"kubernetes.io/projected/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-kube-api-access-9sfwq\") pod \"authorino-6c45d58d97-rxdfp\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:30.044338 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.044270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-tls-cert\") pod \"authorino-6c45d58d97-rxdfp\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:30.062619 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.062574 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlmn\" (UniqueName: \"kubernetes.io/projected/c34bfb3d-06b9-4ff8-ab81-d44577f84b27-kube-api-access-qrlmn\") pod \"authorino-749659456f-j8f2s\" (UID: \"c34bfb3d-06b9-4ff8-ab81-d44577f84b27\") " pod="kuadrant-system/authorino-749659456f-j8f2s" Jul 10 18:11:30.144706 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.144682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfwq\" (UniqueName: \"kubernetes.io/projected/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-kube-api-access-9sfwq\") pod \"authorino-6c45d58d97-rxdfp\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:30.144804 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.144723 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-tls-cert\") pod \"authorino-6c45d58d97-rxdfp\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:30.146982 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.146958 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-tls-cert\") pod \"authorino-6c45d58d97-rxdfp\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:30.154551 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.154534 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfwq\" (UniqueName: \"kubernetes.io/projected/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-kube-api-access-9sfwq\") pod \"authorino-6c45d58d97-rxdfp\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:30.286518 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.286497 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:11:30.426917 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.426847 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c45d58d97-rxdfp"] Jul 10 18:11:30.429622 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:11:30.429569 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58d0ef1e_6725_4384_a7d4_24fa33d2cb30.slice/crio-d6b4478d0a6b4ae3aab72af2cbd150b42e7fff4485d97a1d2c9a92e2db1881a9 WatchSource:0}: Error finding container d6b4478d0a6b4ae3aab72af2cbd150b42e7fff4485d97a1d2c9a92e2db1881a9: Status 404 returned error can't find the container with id d6b4478d0a6b4ae3aab72af2cbd150b42e7fff4485d97a1d2c9a92e2db1881a9 Jul 10 18:11:30.907338 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.907308 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" event={"ID":"58d0ef1e-6725-4384-a7d4-24fa33d2cb30","Type":"ContainerStarted","Data":"69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a"} Jul 10 18:11:30.907740 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.907350 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" event={"ID":"58d0ef1e-6725-4384-a7d4-24fa33d2cb30","Type":"ContainerStarted","Data":"d6b4478d0a6b4ae3aab72af2cbd150b42e7fff4485d97a1d2c9a92e2db1881a9"} Jul 10 18:11:30.909006 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.908975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" event={"ID":"c9b4defc-5343-456d-82ba-bc16c7eee1cf","Type":"ContainerStarted","Data":"99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82"} Jul 10 18:11:30.909106 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.909014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" event={"ID":"c9b4defc-5343-456d-82ba-bc16c7eee1cf","Type":"ContainerStarted","Data":"8645aa3f78e44ec284c7d134f46fed99796b073d94d56019f25d9f47e7eced6f"} Jul 10 18:11:30.909106 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.909020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749659456f-j8f2s" Jul 10 18:11:30.909188 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.909016 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" podUID="c9b4defc-5343-456d-82ba-bc16c7eee1cf" containerName="authorino" containerID="cri-o://99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82" gracePeriod=30 Jul 10 18:11:30.916558 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.916536 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749659456f-j8f2s" Jul 10 18:11:30.927315 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.927267 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" podStartSLOduration=1.5255293540000001 podStartE2EDuration="1.927250521s" totalImagesPullingTime="401.721167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:11:30.926052987 +0000 UTC m=+500.021243475" watchObservedRunningTime="2026-07-10 18:11:30.927250521 +0000 UTC m=+500.022441058" Jul 10 18:11:30.942262 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.942208 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" podStartSLOduration=1.5613856350000002 podStartE2EDuration="1.942188478s" totalImagesPullingTime="380.802843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:11:30.941822144 +0000 UTC m=+500.037012630" watchObservedRunningTime="2026-07-10 18:11:30.942188478 +0000 UTC m=+500.037378965" Jul 10 18:11:30.960162 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.960131 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lnjk6"] Jul 10 18:11:30.960359 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:30.960336 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lnjk6" podUID="e2875c77-3782-4653-af42-f0c089216740" containerName="authorino" containerID="cri-o://33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77" gracePeriod=30 Jul 10 18:11:31.052657 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.052593 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrlmn\" (UniqueName: \"kubernetes.io/projected/c34bfb3d-06b9-4ff8-ab81-d44577f84b27-kube-api-access-qrlmn\") pod \"c34bfb3d-06b9-4ff8-ab81-d44577f84b27\" (UID: \"c34bfb3d-06b9-4ff8-ab81-d44577f84b27\") " Jul 10 18:11:31.054820 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.054794 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34bfb3d-06b9-4ff8-ab81-d44577f84b27-kube-api-access-qrlmn" (OuterVolumeSpecName: "kube-api-access-qrlmn") pod "c34bfb3d-06b9-4ff8-ab81-d44577f84b27" (UID: "c34bfb3d-06b9-4ff8-ab81-d44577f84b27"). InnerVolumeSpecName "kube-api-access-qrlmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:31.154317 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.154281 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrlmn\" (UniqueName: \"kubernetes.io/projected/c34bfb3d-06b9-4ff8-ab81-d44577f84b27-kube-api-access-qrlmn\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:11:31.200058 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.200031 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" Jul 10 18:11:31.230790 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.230764 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lnjk6" Jul 10 18:11:31.356512 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.356473 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zckkq\" (UniqueName: \"kubernetes.io/projected/e2875c77-3782-4653-af42-f0c089216740-kube-api-access-zckkq\") pod \"e2875c77-3782-4653-af42-f0c089216740\" (UID: \"e2875c77-3782-4653-af42-f0c089216740\") " Jul 10 18:11:31.356692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.356536 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdpdn\" (UniqueName: \"kubernetes.io/projected/c9b4defc-5343-456d-82ba-bc16c7eee1cf-kube-api-access-kdpdn\") pod \"c9b4defc-5343-456d-82ba-bc16c7eee1cf\" (UID: \"c9b4defc-5343-456d-82ba-bc16c7eee1cf\") " Jul 10 18:11:31.358833 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.358798 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b4defc-5343-456d-82ba-bc16c7eee1cf-kube-api-access-kdpdn" (OuterVolumeSpecName: "kube-api-access-kdpdn") pod "c9b4defc-5343-456d-82ba-bc16c7eee1cf" (UID: "c9b4defc-5343-456d-82ba-bc16c7eee1cf"). InnerVolumeSpecName "kube-api-access-kdpdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:31.358934 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.358864 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2875c77-3782-4653-af42-f0c089216740-kube-api-access-zckkq" (OuterVolumeSpecName: "kube-api-access-zckkq") pod "e2875c77-3782-4653-af42-f0c089216740" (UID: "e2875c77-3782-4653-af42-f0c089216740"). InnerVolumeSpecName "kube-api-access-zckkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:31.457396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.457367 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zckkq\" (UniqueName: \"kubernetes.io/projected/e2875c77-3782-4653-af42-f0c089216740-kube-api-access-zckkq\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:11:31.457396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.457395 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpdn\" (UniqueName: \"kubernetes.io/projected/c9b4defc-5343-456d-82ba-bc16c7eee1cf-kube-api-access-kdpdn\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:11:31.915231 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.915193 2570 generic.go:356] "Generic (PLEG): container finished" podID="e2875c77-3782-4653-af42-f0c089216740" containerID="33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77" exitCode=0 Jul 10 18:11:31.915694 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.915255 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lnjk6" Jul 10 18:11:31.915694 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.915275 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lnjk6" event={"ID":"e2875c77-3782-4653-af42-f0c089216740","Type":"ContainerDied","Data":"33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77"} Jul 10 18:11:31.915694 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.915323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lnjk6" event={"ID":"e2875c77-3782-4653-af42-f0c089216740","Type":"ContainerDied","Data":"c58cc8309475eaa1abfce840c22b699ccac6b4a31175ddbecdcc9bf17f7310f9"} Jul 10 18:11:31.915694 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.915341 2570 scope.go:117] "RemoveContainer" containerID="33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77" Jul 10 18:11:31.916738 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.916626 2570 generic.go:356] "Generic (PLEG): container finished" podID="c9b4defc-5343-456d-82ba-bc16c7eee1cf" containerID="99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82" exitCode=0 Jul 10 18:11:31.916738 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.916684 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" Jul 10 18:11:31.916738 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.916716 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" event={"ID":"c9b4defc-5343-456d-82ba-bc16c7eee1cf","Type":"ContainerDied","Data":"99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82"} Jul 10 18:11:31.916901 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.916745 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ss2lx" event={"ID":"c9b4defc-5343-456d-82ba-bc16c7eee1cf","Type":"ContainerDied","Data":"8645aa3f78e44ec284c7d134f46fed99796b073d94d56019f25d9f47e7eced6f"} Jul 10 18:11:31.917366 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.917224 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749659456f-j8f2s" Jul 10 18:11:31.924468 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.924450 2570 scope.go:117] "RemoveContainer" containerID="33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77" Jul 10 18:11:31.924753 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:11:31.924734 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77\": container with ID starting with 33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77 not found: ID does not exist" containerID="33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77" Jul 10 18:11:31.924846 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.924777 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77"} err="failed to get container status \"33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77\": rpc error: code = NotFound desc = could not find container \"33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77\": container with ID starting with 33128ff24201c0757aeb80f7c1d3185683457ad1daee66f5a589d937f545cc77 not found: ID does not exist" Jul 10 18:11:31.924846 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.924801 2570 scope.go:117] "RemoveContainer" containerID="99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82" Jul 10 18:11:31.932528 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.932506 2570 scope.go:117] "RemoveContainer" containerID="99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82" Jul 10 18:11:31.932797 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:11:31.932778 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82\": container with ID starting with 99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82 not found: ID does not exist" containerID="99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82" Jul 10 18:11:31.932849 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.932806 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82"} err="failed to get container status \"99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82\": rpc error: code = NotFound desc = could not find container \"99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82\": container with ID starting with 99504875f5aa5d1a08fab6a177194705c8ceb7f2b3b3a37b3d13757166cb4a82 not found: ID does not exist" Jul 10 18:11:31.938393 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.938371 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lnjk6"] Jul 10 18:11:31.944271 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.944252 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lnjk6"] Jul 10 18:11:31.956699 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.956676 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ss2lx"] Jul 10 18:11:31.961514 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.961492 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ss2lx"] Jul 10 18:11:31.991033 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.991003 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-749659456f-j8f2s"] Jul 10 18:11:31.994185 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:31.994158 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-749659456f-j8f2s"] Jul 10 18:11:33.463349 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:33.463313 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c34bfb3d-06b9-4ff8-ab81-d44577f84b27" path="/var/lib/kubelet/pods/c34bfb3d-06b9-4ff8-ab81-d44577f84b27/volumes" Jul 10 18:11:33.463742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:33.463533 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b4defc-5343-456d-82ba-bc16c7eee1cf" path="/var/lib/kubelet/pods/c9b4defc-5343-456d-82ba-bc16c7eee1cf/volumes" Jul 10 18:11:33.463873 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:33.463858 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2875c77-3782-4653-af42-f0c089216740" path="/var/lib/kubelet/pods/e2875c77-3782-4653-af42-f0c089216740/volumes" Jul 10 18:11:35.810894 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.810863 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d4959c5f7-btpbp"] Jul 10 18:11:35.812032 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.812005 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2875c77-3782-4653-af42-f0c089216740" containerName="authorino" Jul 10 18:11:35.812166 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.812155 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2875c77-3782-4653-af42-f0c089216740" containerName="authorino" Jul 10 18:11:35.812279 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.812269 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b4defc-5343-456d-82ba-bc16c7eee1cf" containerName="authorino" Jul 10 18:11:35.812350 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.812342 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b4defc-5343-456d-82ba-bc16c7eee1cf" containerName="authorino" Jul 10 18:11:35.812587 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.812575 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2875c77-3782-4653-af42-f0c089216740" containerName="authorino" Jul 10 18:11:35.812707 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.812696 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b4defc-5343-456d-82ba-bc16c7eee1cf" containerName="authorino" Jul 10 18:11:35.844238 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.844216 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d4959c5f7-btpbp"] Jul 10 18:11:35.844334 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.844296 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:35.847282 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.847252 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 18:11:35.847373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.847285 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vzgps\"" Jul 10 18:11:35.847373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.847252 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 18:11:35.993761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.993733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14e04b30-eb34-492c-8244-3a220b3381e1-webhook-cert\") pod \"maas-controller-d4959c5f7-btpbp\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:35.993871 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:35.993769 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmvk\" (UniqueName: \"kubernetes.io/projected/14e04b30-eb34-492c-8244-3a220b3381e1-kube-api-access-msmvk\") pod \"maas-controller-d4959c5f7-btpbp\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:36.095076 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:36.095013 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14e04b30-eb34-492c-8244-3a220b3381e1-webhook-cert\") pod \"maas-controller-d4959c5f7-btpbp\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:36.095076 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:36.095056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmvk\" (UniqueName: \"kubernetes.io/projected/14e04b30-eb34-492c-8244-3a220b3381e1-kube-api-access-msmvk\") pod \"maas-controller-d4959c5f7-btpbp\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:36.097497 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:36.097477 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14e04b30-eb34-492c-8244-3a220b3381e1-webhook-cert\") pod \"maas-controller-d4959c5f7-btpbp\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:36.104142 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:36.104121 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmvk\" (UniqueName: \"kubernetes.io/projected/14e04b30-eb34-492c-8244-3a220b3381e1-kube-api-access-msmvk\") pod \"maas-controller-d4959c5f7-btpbp\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:36.153543 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:36.153519 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:36.300315 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:36.300291 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d4959c5f7-btpbp"] Jul 10 18:11:36.300453 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:11:36.300430 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e04b30_eb34_492c_8244_3a220b3381e1.slice/crio-5b705e24f77b8f9e86f571c79681ff9ad7b2951a8506beee47caf969ccb55d43 WatchSource:0}: Error finding container 5b705e24f77b8f9e86f571c79681ff9ad7b2951a8506beee47caf969ccb55d43: Status 404 returned error can't find the container with id 5b705e24f77b8f9e86f571c79681ff9ad7b2951a8506beee47caf969ccb55d43 Jul 10 18:11:36.946332 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:36.946292 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4959c5f7-btpbp" event={"ID":"14e04b30-eb34-492c-8244-3a220b3381e1","Type":"ContainerStarted","Data":"5b705e24f77b8f9e86f571c79681ff9ad7b2951a8506beee47caf969ccb55d43"} Jul 10 18:11:38.958530 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:38.958495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4959c5f7-btpbp" event={"ID":"14e04b30-eb34-492c-8244-3a220b3381e1","Type":"ContainerStarted","Data":"c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8"} Jul 10 18:11:38.958941 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:38.958625 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:38.980334 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:38.980280 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d4959c5f7-btpbp" podStartSLOduration=1.9036674310000001 podStartE2EDuration="3.980262755s" totalImagesPullingTime="2.076595324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:11:38.97915933 +0000 UTC m=+508.074349812" watchObservedRunningTime="2026-07-10 18:11:38.980262755 +0000 UTC m=+508.075453241" Jul 10 18:11:49.974967 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:49.974935 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:50.100075 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.100041 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d4959c5f7-btpbp"] Jul 10 18:11:50.100271 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.100249 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d4959c5f7-btpbp" podUID="14e04b30-eb34-492c-8244-3a220b3381e1" containerName="manager" containerID="cri-o://c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8" gracePeriod=30 Jul 10 18:11:50.341986 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.341961 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:50.510244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.510211 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msmvk\" (UniqueName: \"kubernetes.io/projected/14e04b30-eb34-492c-8244-3a220b3381e1-kube-api-access-msmvk\") pod \"14e04b30-eb34-492c-8244-3a220b3381e1\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " Jul 10 18:11:50.510416 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.510296 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14e04b30-eb34-492c-8244-3a220b3381e1-webhook-cert\") pod \"14e04b30-eb34-492c-8244-3a220b3381e1\" (UID: \"14e04b30-eb34-492c-8244-3a220b3381e1\") " Jul 10 18:11:50.512554 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.512519 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e04b30-eb34-492c-8244-3a220b3381e1-kube-api-access-msmvk" (OuterVolumeSpecName: "kube-api-access-msmvk") pod "14e04b30-eb34-492c-8244-3a220b3381e1" (UID: "14e04b30-eb34-492c-8244-3a220b3381e1"). InnerVolumeSpecName "kube-api-access-msmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:50.512693 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.512592 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e04b30-eb34-492c-8244-3a220b3381e1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "14e04b30-eb34-492c-8244-3a220b3381e1" (UID: "14e04b30-eb34-492c-8244-3a220b3381e1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:11:50.611441 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.611373 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmvk\" (UniqueName: \"kubernetes.io/projected/14e04b30-eb34-492c-8244-3a220b3381e1-kube-api-access-msmvk\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:11:50.611441 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:50.611398 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14e04b30-eb34-492c-8244-3a220b3381e1-webhook-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:11:51.018939 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.018901 2570 generic.go:356] "Generic (PLEG): container finished" podID="14e04b30-eb34-492c-8244-3a220b3381e1" containerID="c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8" exitCode=0 Jul 10 18:11:51.019355 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.018948 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4959c5f7-btpbp" event={"ID":"14e04b30-eb34-492c-8244-3a220b3381e1","Type":"ContainerDied","Data":"c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8"} Jul 10 18:11:51.019355 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.018992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d4959c5f7-btpbp" event={"ID":"14e04b30-eb34-492c-8244-3a220b3381e1","Type":"ContainerDied","Data":"5b705e24f77b8f9e86f571c79681ff9ad7b2951a8506beee47caf969ccb55d43"} Jul 10 18:11:51.019355 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.019013 2570 scope.go:117] "RemoveContainer" containerID="c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8" Jul 10 18:11:51.019355 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.018952 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d4959c5f7-btpbp" Jul 10 18:11:51.028571 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.028547 2570 scope.go:117] "RemoveContainer" containerID="c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8" Jul 10 18:11:51.028924 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:11:51.028905 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8\": container with ID starting with c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8 not found: ID does not exist" containerID="c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8" Jul 10 18:11:51.029008 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.028934 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8"} err="failed to get container status \"c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8\": rpc error: code = NotFound desc = could not find container \"c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8\": container with ID starting with c7d6b9f63627c6e51a34a64099f2d3cd58dc9e70bc580b62f71f8b5a34a130e8 not found: ID does not exist" Jul 10 18:11:51.045580 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.045554 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d4959c5f7-btpbp"] Jul 10 18:11:51.048441 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.048420 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d4959c5f7-btpbp"] Jul 10 18:11:51.464068 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:11:51.464033 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e04b30-eb34-492c-8244-3a220b3381e1" path="/var/lib/kubelet/pods/14e04b30-eb34-492c-8244-3a220b3381e1/volumes" Jul 10 18:12:14.803235 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.803201 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-gmms5"] Jul 10 18:12:14.803672 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.803493 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e04b30-eb34-492c-8244-3a220b3381e1" containerName="manager" Jul 10 18:12:14.803672 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.803503 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e04b30-eb34-492c-8244-3a220b3381e1" containerName="manager" Jul 10 18:12:14.803672 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.803564 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e04b30-eb34-492c-8244-3a220b3381e1" containerName="manager" Jul 10 18:12:14.806374 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.806356 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:14.809468 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.809449 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 18:12:14.809962 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.809947 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-knrcw\"" Jul 10 18:12:14.826879 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.826850 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-gmms5"] Jul 10 18:12:14.897026 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.896998 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08f902-e491-4458-ab6c-aade2b0e69d1-plugins-config-volume\") pod \"payload-pre-processing-7964457696-gmms5\" (UID: \"6c08f902-e491-4458-ab6c-aade2b0e69d1\") " pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:14.897150 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.897046 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8mr\" (UniqueName: \"kubernetes.io/projected/6c08f902-e491-4458-ab6c-aade2b0e69d1-kube-api-access-xl8mr\") pod \"payload-pre-processing-7964457696-gmms5\" (UID: \"6c08f902-e491-4458-ab6c-aade2b0e69d1\") " pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:14.998217 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.998190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08f902-e491-4458-ab6c-aade2b0e69d1-plugins-config-volume\") pod \"payload-pre-processing-7964457696-gmms5\" (UID: \"6c08f902-e491-4458-ab6c-aade2b0e69d1\") " pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:14.998326 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.998230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8mr\" (UniqueName: \"kubernetes.io/projected/6c08f902-e491-4458-ab6c-aade2b0e69d1-kube-api-access-xl8mr\") pod \"payload-pre-processing-7964457696-gmms5\" (UID: \"6c08f902-e491-4458-ab6c-aade2b0e69d1\") " pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:14.998872 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:14.998854 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08f902-e491-4458-ab6c-aade2b0e69d1-plugins-config-volume\") pod \"payload-pre-processing-7964457696-gmms5\" (UID: \"6c08f902-e491-4458-ab6c-aade2b0e69d1\") " pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:15.014248 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:15.014223 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8mr\" (UniqueName: \"kubernetes.io/projected/6c08f902-e491-4458-ab6c-aade2b0e69d1-kube-api-access-xl8mr\") pod \"payload-pre-processing-7964457696-gmms5\" (UID: \"6c08f902-e491-4458-ab6c-aade2b0e69d1\") " pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:15.116945 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:15.116891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:15.254869 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:15.254753 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-gmms5"] Jul 10 18:12:15.256909 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:12:15.256885 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c08f902_e491_4458_ab6c_aade2b0e69d1.slice/crio-a7beded37698e786a7144c0a7c8fa3246d35491846c11e5b03e945003a7626f9 WatchSource:0}: Error finding container a7beded37698e786a7144c0a7c8fa3246d35491846c11e5b03e945003a7626f9: Status 404 returned error can't find the container with id a7beded37698e786a7144c0a7c8fa3246d35491846c11e5b03e945003a7626f9 Jul 10 18:12:16.138867 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:16.138828 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" event={"ID":"6c08f902-e491-4458-ab6c-aade2b0e69d1","Type":"ContainerStarted","Data":"a7beded37698e786a7144c0a7c8fa3246d35491846c11e5b03e945003a7626f9"} Jul 10 18:12:17.145529 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:17.145430 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" event={"ID":"6c08f902-e491-4458-ab6c-aade2b0e69d1","Type":"ContainerStarted","Data":"ccc1cdb137635afd7f5c4dff94ba6828b3e9ca7f2fcc39fd25fcc483d439e6f1"} Jul 10 18:12:17.145878 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:17.145532 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:17.167293 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:17.167241 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" podStartSLOduration=1.731804109 podStartE2EDuration="3.16722095s" totalImagesPullingTime="1.435416841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:12:17.164265137 +0000 UTC m=+546.259455623" watchObservedRunningTime="2026-07-10 18:12:17.16722095 +0000 UTC m=+546.262411439" Jul 10 18:12:28.153155 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:28.153121 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-gmms5" Jul 10 18:12:48.873644 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.873574 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh"] Jul 10 18:12:48.877479 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.877454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:48.880280 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.880254 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 18:12:48.881521 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.881498 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 18:12:48.881667 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.881506 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t2lwp\"" Jul 10 18:12:48.881667 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.881505 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 18:12:48.886865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.886844 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh"] Jul 10 18:12:48.973454 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.973411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:48.973705 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.973467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:48.973705 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.973558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5v8\" (UniqueName: \"kubernetes.io/projected/f0ce4e45-4bea-4309-824d-17a4d65f93c5-kube-api-access-rf5v8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:48.973705 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.973674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:48.973705 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.973701 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ce4e45-4bea-4309-824d-17a4d65f93c5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:48.973879 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:48.973744 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.074987 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.074940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.074987 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.074983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ce4e45-4bea-4309-824d-17a4d65f93c5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.075246 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.075106 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.075246 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.075165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.075246 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.075205 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.075246 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.075240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5v8\" (UniqueName: \"kubernetes.io/projected/f0ce4e45-4bea-4309-824d-17a4d65f93c5-kube-api-access-rf5v8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.075477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.075451 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.075529 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.075493 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.075573 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.075537 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.077546 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.077525 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ce4e45-4bea-4309-824d-17a4d65f93c5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.077852 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.077832 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ce4e45-4bea-4309-824d-17a4d65f93c5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.085318 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.085295 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5v8\" (UniqueName: \"kubernetes.io/projected/f0ce4e45-4bea-4309-824d-17a4d65f93c5-kube-api-access-rf5v8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh\" (UID: \"f0ce4e45-4bea-4309-824d-17a4d65f93c5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.190929 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.190895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:49.323352 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:49.323326 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh"] Jul 10 18:12:49.324961 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:12:49.324930 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ce4e45_4bea_4309_824d_17a4d65f93c5.slice/crio-8d579f3cb01aa374a404bf61ee3107a7ad5d0ad37612907e10da39a459f5fa52 WatchSource:0}: Error finding container 8d579f3cb01aa374a404bf61ee3107a7ad5d0ad37612907e10da39a459f5fa52: Status 404 returned error can't find the container with id 8d579f3cb01aa374a404bf61ee3107a7ad5d0ad37612907e10da39a459f5fa52 Jul 10 18:12:50.306488 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:50.306445 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" event={"ID":"f0ce4e45-4bea-4309-824d-17a4d65f93c5","Type":"ContainerStarted","Data":"8d579f3cb01aa374a404bf61ee3107a7ad5d0ad37612907e10da39a459f5fa52"} Jul 10 18:12:51.312509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:51.312461 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" event={"ID":"f0ce4e45-4bea-4309-824d-17a4d65f93c5","Type":"ContainerStarted","Data":"7829fa5a48dd3b709d2ad747238ea896c4b47a7a620ca425b99a714f53b6799c"} Jul 10 18:12:51.335085 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:51.335018 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" podStartSLOduration=2.161942609 podStartE2EDuration="3.334997219s" totalImagesPullingTime="1.17305461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:12:51.332562223 +0000 UTC m=+580.427752710" watchObservedRunningTime="2026-07-10 18:12:51.334997219 +0000 UTC m=+580.430187705" Jul 10 18:12:59.191673 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:59.191629 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:59.192047 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:59.191709 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:59.204252 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:59.204221 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:12:59.361675 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:12:59.361648 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh" Jul 10 18:13:11.371973 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:11.371938 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:13:11.372463 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:11.372442 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:13:15.900759 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.900706 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7"] Jul 10 18:13:15.903168 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.903150 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:15.905987 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.905963 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 18:13:15.915933 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.915906 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7"] Jul 10 18:13:15.997350 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.997317 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvt2w\" (UniqueName: \"kubernetes.io/projected/fbb85468-134c-4f00-a735-b2f4e838dff0-kube-api-access-jvt2w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:15.997350 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.997355 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb85468-134c-4f00-a735-b2f4e838dff0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:15.997568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.997376 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:15.997568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.997465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:15.997568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.997511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:15.997568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:15.997535 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.098406 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvt2w\" (UniqueName: \"kubernetes.io/projected/fbb85468-134c-4f00-a735-b2f4e838dff0-kube-api-access-jvt2w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.098406 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb85468-134c-4f00-a735-b2f4e838dff0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.098679 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.098679 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.098679 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098500 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.098679 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.098952 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.099020 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.098967 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.099075 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.099023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.100993 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.100972 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb85468-134c-4f00-a735-b2f4e838dff0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.101144 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.101127 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb85468-134c-4f00-a735-b2f4e838dff0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.107492 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.107463 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvt2w\" (UniqueName: \"kubernetes.io/projected/fbb85468-134c-4f00-a735-b2f4e838dff0-kube-api-access-jvt2w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7\" (UID: \"fbb85468-134c-4f00-a735-b2f4e838dff0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.214310 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.214269 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:16.351661 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.351632 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7"] Jul 10 18:13:16.353516 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:13:16.353485 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb85468_134c_4f00_a735_b2f4e838dff0.slice/crio-120f0aa17856ee2ffbff8d624be59f94d8e075e9f3d1a551f8fa289bda4e45d4 WatchSource:0}: Error finding container 120f0aa17856ee2ffbff8d624be59f94d8e075e9f3d1a551f8fa289bda4e45d4: Status 404 returned error can't find the container with id 120f0aa17856ee2ffbff8d624be59f94d8e075e9f3d1a551f8fa289bda4e45d4 Jul 10 18:13:16.435315 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:16.435278 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" event={"ID":"fbb85468-134c-4f00-a735-b2f4e838dff0","Type":"ContainerStarted","Data":"120f0aa17856ee2ffbff8d624be59f94d8e075e9f3d1a551f8fa289bda4e45d4"} Jul 10 18:13:17.442014 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:17.441970 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" event={"ID":"fbb85468-134c-4f00-a735-b2f4e838dff0","Type":"ContainerStarted","Data":"f112a9950f8216df43ae27a900e9361dd27e381c9ad141765c310d7d6c1cef48"} Jul 10 18:13:17.464398 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:17.464351 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" podStartSLOduration=2.250906637 podStartE2EDuration="2.464333168s" totalImagesPullingTime="213.426531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:13:17.461408837 +0000 UTC m=+606.556599338" watchObservedRunningTime="2026-07-10 18:13:17.464333168 +0000 UTC m=+606.559523653" Jul 10 18:13:26.215020 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:26.214979 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:26.215666 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:26.215052 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:26.227530 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:26.227503 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:26.501496 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:26.501406 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7" Jul 10 18:13:31.889692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:31.889584 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms"] Jul 10 18:13:31.895278 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:31.895248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:31.897928 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:31.897905 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 18:13:31.904925 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:31.904904 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms"] Jul 10 18:13:32.033026 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.032985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db947848-2de9-4eae-a073-9c007350e923-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.033236 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.033073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6q59\" (UniqueName: \"kubernetes.io/projected/db947848-2de9-4eae-a073-9c007350e923-kube-api-access-j6q59\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.033236 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.033125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.033236 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.033166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.033236 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.033207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.033418 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.033263 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.133849 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.133808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6q59\" (UniqueName: \"kubernetes.io/projected/db947848-2de9-4eae-a073-9c007350e923-kube-api-access-j6q59\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.133849 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.133853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.134107 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.133881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.134107 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.133922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.134107 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.133968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.134221 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.134099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db947848-2de9-4eae-a073-9c007350e923-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.134374 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.134356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.134457 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.134421 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.134518 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.134472 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.136470 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.136438 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db947848-2de9-4eae-a073-9c007350e923-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.136637 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.136620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db947848-2de9-4eae-a073-9c007350e923-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.147915 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.147851 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6q59\" (UniqueName: \"kubernetes.io/projected/db947848-2de9-4eae-a073-9c007350e923-kube-api-access-j6q59\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms\" (UID: \"db947848-2de9-4eae-a073-9c007350e923\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.206913 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.206879 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:32.343722 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.343694 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms"] Jul 10 18:13:32.345047 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:13:32.345013 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb947848_2de9_4eae_a073_9c007350e923.slice/crio-70ff1b5f177884421e23f5086af60270d35fa406a9b81c5fd24f28271db5fa19 WatchSource:0}: Error finding container 70ff1b5f177884421e23f5086af60270d35fa406a9b81c5fd24f28271db5fa19: Status 404 returned error can't find the container with id 70ff1b5f177884421e23f5086af60270d35fa406a9b81c5fd24f28271db5fa19 Jul 10 18:13:32.346453 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.346434 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:13:32.519298 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:32.519251 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" event={"ID":"db947848-2de9-4eae-a073-9c007350e923","Type":"ContainerStarted","Data":"70ff1b5f177884421e23f5086af60270d35fa406a9b81c5fd24f28271db5fa19"} Jul 10 18:13:33.525030 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:33.524993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" event={"ID":"db947848-2de9-4eae-a073-9c007350e923","Type":"ContainerStarted","Data":"fc3cdb1076c1790e2af4dee346910eb8d9bc88f582dfb8c41d91a5bd356a47c1"} Jul 10 18:13:33.544824 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:33.544778 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" podStartSLOduration=2.330412951 podStartE2EDuration="2.544762348s" totalImagesPullingTime="214.349397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:13:33.543528087 +0000 UTC m=+622.638718576" watchObservedRunningTime="2026-07-10 18:13:33.544762348 +0000 UTC m=+622.639952834" Jul 10 18:13:42.207552 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:42.207513 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:42.208012 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:42.207615 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:42.220056 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:42.220027 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:13:42.577828 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:13:42.577741 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms" Jul 10 18:14:30.224636 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.224580 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6789f574f6-mzvs9"] Jul 10 18:14:30.227814 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.227793 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.238949 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.238925 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6789f574f6-mzvs9"] Jul 10 18:14:30.315036 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.315010 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75cc925b-40bd-443f-a84e-45d2ed9bcd17-tls-cert\") pod \"authorino-6789f574f6-mzvs9\" (UID: \"75cc925b-40bd-443f-a84e-45d2ed9bcd17\") " pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.315147 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.315037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb6r\" (UniqueName: \"kubernetes.io/projected/75cc925b-40bd-443f-a84e-45d2ed9bcd17-kube-api-access-ddb6r\") pod \"authorino-6789f574f6-mzvs9\" (UID: \"75cc925b-40bd-443f-a84e-45d2ed9bcd17\") " pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.416476 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.416447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75cc925b-40bd-443f-a84e-45d2ed9bcd17-tls-cert\") pod \"authorino-6789f574f6-mzvs9\" (UID: \"75cc925b-40bd-443f-a84e-45d2ed9bcd17\") " pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.416573 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.416480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb6r\" (UniqueName: \"kubernetes.io/projected/75cc925b-40bd-443f-a84e-45d2ed9bcd17-kube-api-access-ddb6r\") pod \"authorino-6789f574f6-mzvs9\" (UID: \"75cc925b-40bd-443f-a84e-45d2ed9bcd17\") " pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.419178 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.419147 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75cc925b-40bd-443f-a84e-45d2ed9bcd17-tls-cert\") pod \"authorino-6789f574f6-mzvs9\" (UID: \"75cc925b-40bd-443f-a84e-45d2ed9bcd17\") " pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.427412 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.427388 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb6r\" (UniqueName: \"kubernetes.io/projected/75cc925b-40bd-443f-a84e-45d2ed9bcd17-kube-api-access-ddb6r\") pod \"authorino-6789f574f6-mzvs9\" (UID: \"75cc925b-40bd-443f-a84e-45d2ed9bcd17\") " pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.537591 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.537534 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6789f574f6-mzvs9" Jul 10 18:14:30.674113 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.674076 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6789f574f6-mzvs9"] Jul 10 18:14:30.675315 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:14:30.675278 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75cc925b_40bd_443f_a84e_45d2ed9bcd17.slice/crio-bd173f1e70ad3bfdaa2bd3adfd1084d9d497da0b67ccfe7df53b2c5d45ab2c79 WatchSource:0}: Error finding container bd173f1e70ad3bfdaa2bd3adfd1084d9d497da0b67ccfe7df53b2c5d45ab2c79: Status 404 returned error can't find the container with id bd173f1e70ad3bfdaa2bd3adfd1084d9d497da0b67ccfe7df53b2c5d45ab2c79 Jul 10 18:14:30.812431 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:30.812359 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6789f574f6-mzvs9" event={"ID":"75cc925b-40bd-443f-a84e-45d2ed9bcd17","Type":"ContainerStarted","Data":"bd173f1e70ad3bfdaa2bd3adfd1084d9d497da0b67ccfe7df53b2c5d45ab2c79"} Jul 10 18:14:31.820242 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:31.820200 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6789f574f6-mzvs9" event={"ID":"75cc925b-40bd-443f-a84e-45d2ed9bcd17","Type":"ContainerStarted","Data":"c5f96dc8d321462a93851ab80229dd42dd32c043b5eec864d2ad5e61cac14c21"} Jul 10 18:14:31.845477 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:31.845422 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6789f574f6-mzvs9" podStartSLOduration=1.36269963 podStartE2EDuration="1.84540411s" totalImagesPullingTime="482.70448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:14:31.84362864 +0000 UTC m=+680.938819117" watchObservedRunningTime="2026-07-10 18:14:31.84540411 +0000 UTC m=+680.940594596" Jul 10 18:14:31.889858 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:31.889825 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c45d58d97-rxdfp"] Jul 10 18:14:31.890089 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:31.890050 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" podUID="58d0ef1e-6725-4384-a7d4-24fa33d2cb30" containerName="authorino" containerID="cri-o://69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a" gracePeriod=30 Jul 10 18:14:32.155607 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.155566 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:14:32.232859 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.232828 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sfwq\" (UniqueName: \"kubernetes.io/projected/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-kube-api-access-9sfwq\") pod \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " Jul 10 18:14:32.233031 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.232909 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-tls-cert\") pod \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\" (UID: \"58d0ef1e-6725-4384-a7d4-24fa33d2cb30\") " Jul 10 18:14:32.235183 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.235152 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-kube-api-access-9sfwq" (OuterVolumeSpecName: "kube-api-access-9sfwq") pod "58d0ef1e-6725-4384-a7d4-24fa33d2cb30" (UID: "58d0ef1e-6725-4384-a7d4-24fa33d2cb30"). InnerVolumeSpecName "kube-api-access-9sfwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:14:32.243956 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.243931 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "58d0ef1e-6725-4384-a7d4-24fa33d2cb30" (UID: "58d0ef1e-6725-4384-a7d4-24fa33d2cb30"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:14:32.333804 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.333764 2570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-tls-cert\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:14:32.333804 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.333792 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfwq\" (UniqueName: \"kubernetes.io/projected/58d0ef1e-6725-4384-a7d4-24fa33d2cb30-kube-api-access-9sfwq\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:14:32.827292 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.827198 2570 generic.go:356] "Generic (PLEG): container finished" podID="58d0ef1e-6725-4384-a7d4-24fa33d2cb30" containerID="69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a" exitCode=0 Jul 10 18:14:32.827292 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.827244 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" Jul 10 18:14:32.827722 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.827293 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" event={"ID":"58d0ef1e-6725-4384-a7d4-24fa33d2cb30","Type":"ContainerDied","Data":"69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a"} Jul 10 18:14:32.827722 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.827340 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c45d58d97-rxdfp" event={"ID":"58d0ef1e-6725-4384-a7d4-24fa33d2cb30","Type":"ContainerDied","Data":"d6b4478d0a6b4ae3aab72af2cbd150b42e7fff4485d97a1d2c9a92e2db1881a9"} Jul 10 18:14:32.827722 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.827358 2570 scope.go:117] "RemoveContainer" containerID="69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a" Jul 10 18:14:32.836547 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.836527 2570 scope.go:117] "RemoveContainer" containerID="69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a" Jul 10 18:14:32.836854 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:14:32.836828 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a\": container with ID starting with 69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a not found: ID does not exist" containerID="69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a" Jul 10 18:14:32.836918 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.836869 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a"} err="failed to get container status \"69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a\": rpc error: code = NotFound desc = could not find container \"69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a\": container with ID starting with 69c74b5a1b7e42a19428b09c936c21abcf60e4c5a11e6969397089951092ed1a not found: ID does not exist" Jul 10 18:14:32.856556 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.856535 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c45d58d97-rxdfp"] Jul 10 18:14:32.866631 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:32.866606 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c45d58d97-rxdfp"] Jul 10 18:14:33.463778 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:14:33.463744 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d0ef1e-6725-4384-a7d4-24fa33d2cb30" path="/var/lib/kubelet/pods/58d0ef1e-6725-4384-a7d4-24fa33d2cb30/volumes" Jul 10 18:18:11.403229 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:18:11.403151 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:18:11.404513 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:18:11.404492 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:23:11.440872 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:23:11.440838 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:23:11.443915 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:23:11.443893 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:28:11.475681 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:28:11.475649 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:28:11.479531 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:28:11.479508 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:33:11.507204 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:33:11.507178 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:33:11.513122 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:33:11.513101 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:34:44.388991 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.388951 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh"] Jul 10 18:34:44.389509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.389262 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d0ef1e-6725-4384-a7d4-24fa33d2cb30" containerName="authorino" Jul 10 18:34:44.389509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.389272 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d0ef1e-6725-4384-a7d4-24fa33d2cb30" containerName="authorino" Jul 10 18:34:44.389509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.389342 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d0ef1e-6725-4384-a7d4-24fa33d2cb30" containerName="authorino" Jul 10 18:34:44.392358 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.392339 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.407070 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.407049 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tls-certs\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.407200 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.407084 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tmp-dir\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.407200 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.407107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw5ck\" (UniqueName: \"kubernetes.io/projected/634d24b1-df6e-4fa4-84af-a77eb1218dbe-kube-api-access-pw5ck\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.407200 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.407125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-dshm\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.407307 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.407213 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-model-cache\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.407307 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.407255 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-home\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.409755 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.409734 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7d2779\"/\"openshift-service-ca.crt\"" Jul 10 18:34:44.409822 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.409761 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7d2779\"/\"kube-root-ca.crt\"" Jul 10 18:34:44.409822 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.409800 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7d2779\"/\"test-model-7d2779-kserve-self-signed-certs\"" Jul 10 18:34:44.410848 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.410835 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7d2779\"/\"default-dockercfg-cqxw4\"" Jul 10 18:34:44.458787 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.458757 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh"] Jul 10 18:34:44.508075 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508043 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tmp-dir\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508087 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw5ck\" (UniqueName: \"kubernetes.io/projected/634d24b1-df6e-4fa4-84af-a77eb1218dbe-kube-api-access-pw5ck\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-dshm\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-model-cache\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-home\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508498 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tls-certs\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508553 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508510 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tmp-dir\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508553 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508540 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-model-cache\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.508691 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.508645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-home\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.510814 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.510791 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-dshm\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.510958 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.510941 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tls-certs\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.535088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.535059 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw5ck\" (UniqueName: \"kubernetes.io/projected/634d24b1-df6e-4fa4-84af-a77eb1218dbe-kube-api-access-pw5ck\") pod \"test-model-7d2779-kserve-669bf8c577-vgwlh\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.703499 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.703460 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:44.888026 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:34:44.887994 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod634d24b1_df6e_4fa4_84af_a77eb1218dbe.slice/crio-8920b204ce81027db5e57988266de84446303386c6b5c65ecac518f42e396a23 WatchSource:0}: Error finding container 8920b204ce81027db5e57988266de84446303386c6b5c65ecac518f42e396a23: Status 404 returned error can't find the container with id 8920b204ce81027db5e57988266de84446303386c6b5c65ecac518f42e396a23 Jul 10 18:34:44.889288 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:44.889263 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh"] Jul 10 18:34:45.894537 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:45.894497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" event={"ID":"634d24b1-df6e-4fa4-84af-a77eb1218dbe","Type":"ContainerStarted","Data":"13c0914a109d1e6e8940feb96793524efc493e85db95f052446e32fa0492ee8c"} Jul 10 18:34:45.894537 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:45.894541 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" event={"ID":"634d24b1-df6e-4fa4-84af-a77eb1218dbe","Type":"ContainerStarted","Data":"8920b204ce81027db5e57988266de84446303386c6b5c65ecac518f42e396a23"} Jul 10 18:34:45.951842 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:45.951790 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" podStartSLOduration=1.9517742710000001 podStartE2EDuration="1.951774271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:34:45.947703896 +0000 UTC m=+1895.042894381" watchObservedRunningTime="2026-07-10 18:34:45.951774271 +0000 UTC m=+1895.046964757" Jul 10 18:34:54.704034 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:54.703998 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:54.704462 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:54.704053 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:54.716368 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:54.716342 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:34:54.951358 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:34:54.951331 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:35:48.149124 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:35:48.149089 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh"] Jul 10 18:36:03.169568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.169467 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" podUID="634d24b1-df6e-4fa4-84af-a77eb1218dbe" containerName="main" containerID="cri-o://13c0914a109d1e6e8940feb96793524efc493e85db95f052446e32fa0492ee8c" gracePeriod=45 Jul 10 18:36:03.286101 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.286073 2570 generic.go:356] "Generic (PLEG): container finished" podID="634d24b1-df6e-4fa4-84af-a77eb1218dbe" containerID="13c0914a109d1e6e8940feb96793524efc493e85db95f052446e32fa0492ee8c" exitCode=0 Jul 10 18:36:03.286216 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.286111 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" event={"ID":"634d24b1-df6e-4fa4-84af-a77eb1218dbe","Type":"ContainerDied","Data":"13c0914a109d1e6e8940feb96793524efc493e85db95f052446e32fa0492ee8c"} Jul 10 18:36:03.410557 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.410530 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:36:03.505761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.505683 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tls-certs\") pod \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " Jul 10 18:36:03.505892 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.505759 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw5ck\" (UniqueName: \"kubernetes.io/projected/634d24b1-df6e-4fa4-84af-a77eb1218dbe-kube-api-access-pw5ck\") pod \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " Jul 10 18:36:03.505892 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.505861 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-dshm\") pod \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " Jul 10 18:36:03.505974 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.505932 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-model-cache\") pod \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " Jul 10 18:36:03.506029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.505970 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tmp-dir\") pod \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " Jul 10 18:36:03.506029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.506002 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-home\") pod \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\" (UID: \"634d24b1-df6e-4fa4-84af-a77eb1218dbe\") " Jul 10 18:36:03.506200 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.506173 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "634d24b1-df6e-4fa4-84af-a77eb1218dbe" (UID: "634d24b1-df6e-4fa4-84af-a77eb1218dbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:03.506372 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.506271 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "634d24b1-df6e-4fa4-84af-a77eb1218dbe" (UID: "634d24b1-df6e-4fa4-84af-a77eb1218dbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:03.506527 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.506373 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-home" (OuterVolumeSpecName: "home") pod "634d24b1-df6e-4fa4-84af-a77eb1218dbe" (UID: "634d24b1-df6e-4fa4-84af-a77eb1218dbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:03.508145 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.508117 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-dshm" (OuterVolumeSpecName: "dshm") pod "634d24b1-df6e-4fa4-84af-a77eb1218dbe" (UID: "634d24b1-df6e-4fa4-84af-a77eb1218dbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:03.508234 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.508161 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "634d24b1-df6e-4fa4-84af-a77eb1218dbe" (UID: "634d24b1-df6e-4fa4-84af-a77eb1218dbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:36:03.508234 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.508156 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634d24b1-df6e-4fa4-84af-a77eb1218dbe-kube-api-access-pw5ck" (OuterVolumeSpecName: "kube-api-access-pw5ck") pod "634d24b1-df6e-4fa4-84af-a77eb1218dbe" (UID: "634d24b1-df6e-4fa4-84af-a77eb1218dbe"). InnerVolumeSpecName "kube-api-access-pw5ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:36:03.607318 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.607284 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw5ck\" (UniqueName: \"kubernetes.io/projected/634d24b1-df6e-4fa4-84af-a77eb1218dbe-kube-api-access-pw5ck\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:36:03.607318 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.607313 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-dshm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:36:03.607558 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.607327 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-model-cache\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:36:03.607558 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.607340 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tmp-dir\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:36:03.607558 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.607352 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634d24b1-df6e-4fa4-84af-a77eb1218dbe-home\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:36:03.607558 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:03.607363 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634d24b1-df6e-4fa4-84af-a77eb1218dbe-tls-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:36:04.292940 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:04.292903 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" event={"ID":"634d24b1-df6e-4fa4-84af-a77eb1218dbe","Type":"ContainerDied","Data":"8920b204ce81027db5e57988266de84446303386c6b5c65ecac518f42e396a23"} Jul 10 18:36:04.293399 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:04.292962 2570 scope.go:117] "RemoveContainer" containerID="13c0914a109d1e6e8940feb96793524efc493e85db95f052446e32fa0492ee8c" Jul 10 18:36:04.293399 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:04.292913 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh" Jul 10 18:36:04.318250 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:04.318220 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh"] Jul 10 18:36:04.323984 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:04.323958 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7d2779/test-model-7d2779-kserve-669bf8c577-vgwlh"] Jul 10 18:36:05.463739 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:36:05.463707 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634d24b1-df6e-4fa4-84af-a77eb1218dbe" path="/var/lib/kubelet/pods/634d24b1-df6e-4fa4-84af-a77eb1218dbe/volumes" Jul 10 18:38:11.538238 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:11.538211 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:38:11.551725 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:11.551695 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:38:16.194065 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.194033 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb"] Jul 10 18:38:16.194445 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.194357 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634d24b1-df6e-4fa4-84af-a77eb1218dbe" containerName="main" Jul 10 18:38:16.194445 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.194369 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="634d24b1-df6e-4fa4-84af-a77eb1218dbe" containerName="main" Jul 10 18:38:16.194445 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.194433 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="634d24b1-df6e-4fa4-84af-a77eb1218dbe" containerName="main" Jul 10 18:38:16.197188 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.197167 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.200147 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.200115 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"openshift-service-ca.crt\"" Jul 10 18:38:16.201471 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.201277 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"default-dockercfg-s8p4h\"" Jul 10 18:38:16.201471 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.201376 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"auth-test-model-319db5-kserve-self-signed-certs\"" Jul 10 18:38:16.201706 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.201466 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"kube-root-ca.crt\"" Jul 10 18:38:16.208483 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.208454 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb"] Jul 10 18:38:16.240133 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.240107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-home\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.240241 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.240151 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tmp-dir\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.240241 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.240174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tls-certs\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.240313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.240245 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmcm\" (UniqueName: \"kubernetes.io/projected/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-kube-api-access-8fmcm\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.240313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.240286 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-dshm\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.240313 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.240302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-model-cache\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.341517 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.341491 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tmp-dir\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.341664 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.341519 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tls-certs\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.341664 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.341545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmcm\" (UniqueName: \"kubernetes.io/projected/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-kube-api-access-8fmcm\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.341796 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.341684 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-dshm\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.341796 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.341720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-model-cache\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.341967 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.341803 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-home\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.341967 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.341902 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tmp-dir\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.342099 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.342066 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-model-cache\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.342155 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.342119 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-home\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.344077 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.344053 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-dshm\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.344239 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.344222 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tls-certs\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.350190 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.350165 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmcm\" (UniqueName: \"kubernetes.io/projected/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-kube-api-access-8fmcm\") pod \"auth-test-model-319db5-kserve-c765745bc-q8pxb\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.507807 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.507757 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:16.638229 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.638194 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb"] Jul 10 18:38:16.641039 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:38:16.641010 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaafb25db_a67c_4cfe_b52d_91f9ffd3c0a2.slice/crio-9893b348d75f0f7924a41a37ca1019c4ee2ac7f87da4c2a02e5829036d1b527c WatchSource:0}: Error finding container 9893b348d75f0f7924a41a37ca1019c4ee2ac7f87da4c2a02e5829036d1b527c: Status 404 returned error can't find the container with id 9893b348d75f0f7924a41a37ca1019c4ee2ac7f87da4c2a02e5829036d1b527c Jul 10 18:38:16.934751 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.934713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" event={"ID":"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2","Type":"ContainerStarted","Data":"b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7"} Jul 10 18:38:16.934904 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.934757 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" event={"ID":"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2","Type":"ContainerStarted","Data":"9893b348d75f0f7924a41a37ca1019c4ee2ac7f87da4c2a02e5829036d1b527c"} Jul 10 18:38:16.954125 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:16.954077 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" podStartSLOduration=0.954063185 podStartE2EDuration="954.063185ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:38:16.953046928 +0000 UTC m=+2106.048237415" watchObservedRunningTime="2026-07-10 18:38:16.954063185 +0000 UTC m=+2106.049253670" Jul 10 18:38:26.508559 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:26.508525 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:26.509000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:26.508573 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:26.521286 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:26.521257 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:26.997678 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:26.997650 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:38:38.980782 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:38.980749 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp"] Jul 10 18:38:38.984208 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:38.984188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:38.987541 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:38.987519 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"kube-root-ca.crt\"" Jul 10 18:38:38.988761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:38.988738 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"openshift-service-ca.crt\"" Jul 10 18:38:38.988761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:38.988749 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"default-dockercfg-mf8f5\"" Jul 10 18:38:38.988917 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:38.988749 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"auth-test-model-4cfdb5-kserve-self-signed-certs\"" Jul 10 18:38:39.004540 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.004516 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp"] Jul 10 18:38:39.120685 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.120663 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-model-cache\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.120791 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.120696 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-home\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.120791 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.120745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-dshm\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.120874 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.120793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t578r\" (UniqueName: \"kubernetes.io/projected/07cae728-a30d-4f98-867e-47be5af286cf-kube-api-access-t578r\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.120874 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.120847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cae728-a30d-4f98-867e-47be5af286cf-tls-certs\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.120874 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.120868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-tmp-dir\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221301 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-dshm\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221407 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t578r\" (UniqueName: \"kubernetes.io/projected/07cae728-a30d-4f98-867e-47be5af286cf-kube-api-access-t578r\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221407 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cae728-a30d-4f98-867e-47be5af286cf-tls-certs\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221407 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221361 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-tmp-dir\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221544 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-model-cache\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221544 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-home\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221787 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221766 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-tmp-dir\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221879 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221790 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-home\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.221879 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.221833 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-model-cache\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.223805 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.223784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-dshm\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.224054 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.224038 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cae728-a30d-4f98-867e-47be5af286cf-tls-certs\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.233091 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.233041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t578r\" (UniqueName: \"kubernetes.io/projected/07cae728-a30d-4f98-867e-47be5af286cf-kube-api-access-t578r\") pod \"auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.294847 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.294825 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:39.437810 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:39.437787 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp"] Jul 10 18:38:39.439303 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:38:39.439281 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07cae728_a30d_4f98_867e_47be5af286cf.slice/crio-f957b93a8ef853e3376d6ca58696921ce6d4c67850816a8e336a6f941deb6f6b WatchSource:0}: Error finding container f957b93a8ef853e3376d6ca58696921ce6d4c67850816a8e336a6f941deb6f6b: Status 404 returned error can't find the container with id f957b93a8ef853e3376d6ca58696921ce6d4c67850816a8e336a6f941deb6f6b Jul 10 18:38:40.052664 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:40.052627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" event={"ID":"07cae728-a30d-4f98-867e-47be5af286cf","Type":"ContainerStarted","Data":"d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7"} Jul 10 18:38:40.052664 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:40.052675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" event={"ID":"07cae728-a30d-4f98-867e-47be5af286cf","Type":"ContainerStarted","Data":"f957b93a8ef853e3376d6ca58696921ce6d4c67850816a8e336a6f941deb6f6b"} Jul 10 18:38:49.294994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:49.294961 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:49.295515 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:49.295013 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:49.307328 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:49.307303 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:38:49.330070 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:49.330027 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" podStartSLOduration=11.330012251 podStartE2EDuration="11.330012251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:38:40.079642369 +0000 UTC m=+2129.174832849" watchObservedRunningTime="2026-07-10 18:38:49.330012251 +0000 UTC m=+2138.425202775" Jul 10 18:38:50.117884 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:38:50.117855 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:40:35.320434 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:35.320346 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb"] Jul 10 18:40:36.282056 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:36.282014 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp"] Jul 10 18:40:50.341509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.341447 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" podUID="aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" containerName="main" containerID="cri-o://b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7" gracePeriod=45 Jul 10 18:40:50.600975 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.600908 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:40:50.717169 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.717136 2570 generic.go:356] "Generic (PLEG): container finished" podID="aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" containerID="b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7" exitCode=0 Jul 10 18:40:50.717373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.717178 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" Jul 10 18:40:50.717373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.717191 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" event={"ID":"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2","Type":"ContainerDied","Data":"b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7"} Jul 10 18:40:50.717373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.717227 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb" event={"ID":"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2","Type":"ContainerDied","Data":"9893b348d75f0f7924a41a37ca1019c4ee2ac7f87da4c2a02e5829036d1b527c"} Jul 10 18:40:50.717373 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.717245 2570 scope.go:117] "RemoveContainer" containerID="b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7" Jul 10 18:40:50.726036 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.726014 2570 scope.go:117] "RemoveContainer" containerID="b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7" Jul 10 18:40:50.726324 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:40:50.726302 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7\": container with ID starting with b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7 not found: ID does not exist" containerID="b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7" Jul 10 18:40:50.726375 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.726334 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7"} err="failed to get container status \"b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7\": rpc error: code = NotFound desc = could not find container \"b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7\": container with ID starting with b8ff218c669088845b9f17eadae8c8083bac37e152cc86c7a6d368a4144c5ed7 not found: ID does not exist" Jul 10 18:40:50.750611 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.750557 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tls-certs\") pod \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " Jul 10 18:40:50.750757 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.750662 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tmp-dir\") pod \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " Jul 10 18:40:50.750814 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.750750 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-dshm\") pod \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " Jul 10 18:40:50.750814 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.750807 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fmcm\" (UniqueName: \"kubernetes.io/projected/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-kube-api-access-8fmcm\") pod \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " Jul 10 18:40:50.750922 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.750847 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" (UID: "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:50.750922 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.750853 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-home\") pod \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " Jul 10 18:40:50.750922 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.750895 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-model-cache\") pod \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\" (UID: \"aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2\") " Jul 10 18:40:50.751082 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.751043 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-home" (OuterVolumeSpecName: "home") pod "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" (UID: "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:50.751178 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.751146 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" (UID: "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:50.751284 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.751268 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tmp-dir\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:50.751352 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.751285 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-home\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:50.751352 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.751295 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-model-cache\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:50.753101 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.753071 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-dshm" (OuterVolumeSpecName: "dshm") pod "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" (UID: "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:50.753227 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.753099 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-kube-api-access-8fmcm" (OuterVolumeSpecName: "kube-api-access-8fmcm") pod "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" (UID: "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2"). InnerVolumeSpecName "kube-api-access-8fmcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:40:50.753227 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.753196 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" (UID: "aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:40:50.852850 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.852758 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-dshm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:50.852850 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.852793 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fmcm\" (UniqueName: \"kubernetes.io/projected/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-kube-api-access-8fmcm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:50.852850 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.852804 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2-tls-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:50.986848 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.986815 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl"] Jul 10 18:40:50.987169 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.987149 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" containerName="main" Jul 10 18:40:50.987217 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.987169 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" containerName="main" Jul 10 18:40:50.987256 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.987229 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" containerName="main" Jul 10 18:40:50.991471 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.991452 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:50.994038 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:50.994014 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"sub-test-model-319db5-kserve-self-signed-certs\"" Jul 10 18:40:51.000476 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.000449 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl"] Jul 10 18:40:51.041356 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.041315 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb"] Jul 10 18:40:51.048815 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.048787 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/auth-test-model-319db5-kserve-c765745bc-q8pxb"] Jul 10 18:40:51.155553 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.155451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tmp-dir\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.155553 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.155490 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-dshm\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.155553 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.155512 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-model-cache\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.155858 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.155644 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-home\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.155858 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.155704 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tls-certs\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.155858 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.155782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwn5\" (UniqueName: \"kubernetes.io/projected/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-kube-api-access-gnwn5\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.256463 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.256420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwn5\" (UniqueName: \"kubernetes.io/projected/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-kube-api-access-gnwn5\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.256692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.256488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tmp-dir\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.256692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.256510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-dshm\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.256692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.256527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-model-cache\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.256692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.256549 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-home\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.256692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.256587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tls-certs\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.257047 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.257025 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tmp-dir\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.257117 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.257055 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-model-cache\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.257168 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.257152 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-home\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.259209 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.259187 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-dshm\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.259362 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.259346 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tls-certs\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.267458 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.267428 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwn5\" (UniqueName: \"kubernetes.io/projected/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-kube-api-access-gnwn5\") pod \"sub-test-model-319db5-kserve-7746758695-4lrtl\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.301641 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.301576 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:40:51.302808 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.302752 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" podUID="07cae728-a30d-4f98-867e-47be5af286cf" containerName="main" containerID="cri-o://d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7" gracePeriod=45 Jul 10 18:40:51.447242 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.447205 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl"] Jul 10 18:40:51.466572 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.466543 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2" path="/var/lib/kubelet/pods/aafb25db-a67c-4cfe-b52d-91f9ffd3c0a2/volumes" Jul 10 18:40:51.564126 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.564098 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:40:51.723854 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.723758 2570 generic.go:356] "Generic (PLEG): container finished" podID="07cae728-a30d-4f98-867e-47be5af286cf" containerID="d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7" exitCode=0 Jul 10 18:40:51.723854 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.723819 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" Jul 10 18:40:51.724096 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.723842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" event={"ID":"07cae728-a30d-4f98-867e-47be5af286cf","Type":"ContainerDied","Data":"d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7"} Jul 10 18:40:51.724096 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.723892 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp" event={"ID":"07cae728-a30d-4f98-867e-47be5af286cf","Type":"ContainerDied","Data":"f957b93a8ef853e3376d6ca58696921ce6d4c67850816a8e336a6f941deb6f6b"} Jul 10 18:40:51.724096 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.723915 2570 scope.go:117] "RemoveContainer" containerID="d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7" Jul 10 18:40:51.725430 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.725393 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" event={"ID":"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1","Type":"ContainerStarted","Data":"3f2818e663f0f5e9ec32ed88de672b43618906c1c1860bb1f56673a0512fa8e0"} Jul 10 18:40:51.725538 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.725446 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" event={"ID":"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1","Type":"ContainerStarted","Data":"5b0dbe10ba32c5be6f3ab8f9ff0068ce6f30e0137f9ce7de8980955a4ca56d47"} Jul 10 18:40:51.732669 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.732651 2570 scope.go:117] "RemoveContainer" containerID="d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7" Jul 10 18:40:51.732940 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:40:51.732913 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7\": container with ID starting with d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7 not found: ID does not exist" containerID="d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7" Jul 10 18:40:51.733039 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.732945 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7"} err="failed to get container status \"d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7\": rpc error: code = NotFound desc = could not find container \"d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7\": container with ID starting with d61e44c3074a80cfa8f295745c0ff82457f76073d029a93d9819b501503aacd7 not found: ID does not exist" Jul 10 18:40:51.746047 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.745996 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" podStartSLOduration=1.745979766 podStartE2EDuration="1.745979766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:51.744186836 +0000 UTC m=+2260.839377313" watchObservedRunningTime="2026-07-10 18:40:51.745979766 +0000 UTC m=+2260.841170251" Jul 10 18:40:51.762074 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762046 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-model-cache\") pod \"07cae728-a30d-4f98-867e-47be5af286cf\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " Jul 10 18:40:51.762244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762088 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-dshm\") pod \"07cae728-a30d-4f98-867e-47be5af286cf\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " Jul 10 18:40:51.762244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762127 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cae728-a30d-4f98-867e-47be5af286cf-tls-certs\") pod \"07cae728-a30d-4f98-867e-47be5af286cf\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " Jul 10 18:40:51.762244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762152 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-tmp-dir\") pod \"07cae728-a30d-4f98-867e-47be5af286cf\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " Jul 10 18:40:51.762244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762186 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-home\") pod \"07cae728-a30d-4f98-867e-47be5af286cf\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " Jul 10 18:40:51.762244 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762230 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t578r\" (UniqueName: \"kubernetes.io/projected/07cae728-a30d-4f98-867e-47be5af286cf-kube-api-access-t578r\") pod \"07cae728-a30d-4f98-867e-47be5af286cf\" (UID: \"07cae728-a30d-4f98-867e-47be5af286cf\") " Jul 10 18:40:51.762498 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762415 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "07cae728-a30d-4f98-867e-47be5af286cf" (UID: "07cae728-a30d-4f98-867e-47be5af286cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:51.762498 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762449 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07cae728-a30d-4f98-867e-47be5af286cf" (UID: "07cae728-a30d-4f98-867e-47be5af286cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:51.762498 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762465 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-home" (OuterVolumeSpecName: "home") pod "07cae728-a30d-4f98-867e-47be5af286cf" (UID: "07cae728-a30d-4f98-867e-47be5af286cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:51.762652 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.762540 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-model-cache\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:51.764559 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.764536 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-dshm" (OuterVolumeSpecName: "dshm") pod "07cae728-a30d-4f98-867e-47be5af286cf" (UID: "07cae728-a30d-4f98-867e-47be5af286cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:51.764661 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.764570 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cae728-a30d-4f98-867e-47be5af286cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07cae728-a30d-4f98-867e-47be5af286cf" (UID: "07cae728-a30d-4f98-867e-47be5af286cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:40:51.764702 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.764672 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cae728-a30d-4f98-867e-47be5af286cf-kube-api-access-t578r" (OuterVolumeSpecName: "kube-api-access-t578r") pod "07cae728-a30d-4f98-867e-47be5af286cf" (UID: "07cae728-a30d-4f98-867e-47be5af286cf"). InnerVolumeSpecName "kube-api-access-t578r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:40:51.863169 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.863122 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-dshm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:51.863169 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.863167 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cae728-a30d-4f98-867e-47be5af286cf-tls-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:51.863381 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.863182 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-tmp-dir\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:51.863381 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.863193 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cae728-a30d-4f98-867e-47be5af286cf-home\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:51.863381 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:51.863204 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t578r\" (UniqueName: \"kubernetes.io/projected/07cae728-a30d-4f98-867e-47be5af286cf-kube-api-access-t578r\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:40:52.045794 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:52.045759 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp"] Jul 10 18:40:52.048568 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:52.048544 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/auth-test-model-4cfdb5-kserve-7b4f47c7bc-z6xvp"] Jul 10 18:40:53.464036 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:40:53.464000 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07cae728-a30d-4f98-867e-47be5af286cf" path="/var/lib/kubelet/pods/07cae728-a30d-4f98-867e-47be5af286cf/volumes" Jul 10 18:41:01.302685 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:01.302630 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:41:01.302685 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:01.302697 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:41:01.315623 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:01.315565 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:41:01.789161 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:01.789126 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:41:29.018661 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.018629 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj"] Jul 10 18:41:29.019088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.018987 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07cae728-a30d-4f98-867e-47be5af286cf" containerName="main" Jul 10 18:41:29.019088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.018998 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="07cae728-a30d-4f98-867e-47be5af286cf" containerName="main" Jul 10 18:41:29.019088 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.019057 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="07cae728-a30d-4f98-867e-47be5af286cf" containerName="main" Jul 10 18:41:29.022237 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.022215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.025068 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.025045 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"sub-test-model-4cfdb5-kserve-self-signed-certs\"" Jul 10 18:41:29.026248 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.026229 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"kube-root-ca.crt\"" Jul 10 18:41:29.027071 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.027050 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"openshift-service-ca.crt\"" Jul 10 18:41:29.027071 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.027050 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"default-dockercfg-mf8f5\"" Jul 10 18:41:29.052664 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.052631 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj"] Jul 10 18:41:29.105161 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.105121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tls-certs\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.105333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.105166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-model-cache\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.105333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.105214 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tmp-dir\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.105333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.105243 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-dshm\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.105333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.105268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-home\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.105469 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.105356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2s6\" (UniqueName: \"kubernetes.io/projected/9b6de3b9-72f9-420b-bc65-c144e6db40b6-kube-api-access-hz2s6\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.206337 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-dshm\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.206337 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-home\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.206624 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2s6\" (UniqueName: \"kubernetes.io/projected/9b6de3b9-72f9-420b-bc65-c144e6db40b6-kube-api-access-hz2s6\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.206624 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206457 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tls-certs\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.206748 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-model-cache\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.206748 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tmp-dir\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.206947 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-home\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.207034 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.206947 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-model-cache\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.207175 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.207062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tmp-dir\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.208824 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.208801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-dshm\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.209091 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.209075 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tls-certs\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.218128 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.218100 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2s6\" (UniqueName: \"kubernetes.io/projected/9b6de3b9-72f9-420b-bc65-c144e6db40b6-kube-api-access-hz2s6\") pod \"sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.331465 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.331367 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:29.470901 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.470873 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj"] Jul 10 18:41:29.472543 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:41:29.472517 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b6de3b9_72f9_420b_bc65_c144e6db40b6.slice/crio-a9b96f1c1bc6558ca8cc055bd24bd92ff990b362e9b9156c664a0f2372b086ff WatchSource:0}: Error finding container a9b96f1c1bc6558ca8cc055bd24bd92ff990b362e9b9156c664a0f2372b086ff: Status 404 returned error can't find the container with id a9b96f1c1bc6558ca8cc055bd24bd92ff990b362e9b9156c664a0f2372b086ff Jul 10 18:41:29.929804 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.929759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" event={"ID":"9b6de3b9-72f9-420b-bc65-c144e6db40b6","Type":"ContainerStarted","Data":"f32f5a514ebcbfe4574cbd2dd2394142c19a02c084842227324c6707b99f252e"} Jul 10 18:41:29.929987 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.929814 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" event={"ID":"9b6de3b9-72f9-420b-bc65-c144e6db40b6","Type":"ContainerStarted","Data":"a9b96f1c1bc6558ca8cc055bd24bd92ff990b362e9b9156c664a0f2372b086ff"} Jul 10 18:41:29.949796 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:29.949742 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" podStartSLOduration=1.9497227179999999 podStartE2EDuration="1.949722718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:29.947777436 +0000 UTC m=+2299.042967924" watchObservedRunningTime="2026-07-10 18:41:29.949722718 +0000 UTC m=+2299.044913205" Jul 10 18:41:39.332440 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:39.332401 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:39.332877 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:39.332462 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:39.344943 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:39.344916 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:41:39.990992 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:41:39.990961 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:42:06.969533 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:06.969452 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl"] Jul 10 18:42:08.061754 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:08.061721 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj"] Jul 10 18:42:21.990394 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:21.990354 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" podUID="e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" containerName="main" containerID="cri-o://3f2818e663f0f5e9ec32ed88de672b43618906c1c1860bb1f56673a0512fa8e0" gracePeriod=45 Jul 10 18:42:22.196380 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.196337 2570 generic.go:356] "Generic (PLEG): container finished" podID="e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" containerID="3f2818e663f0f5e9ec32ed88de672b43618906c1c1860bb1f56673a0512fa8e0" exitCode=0 Jul 10 18:42:22.196669 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.196469 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" event={"ID":"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1","Type":"ContainerDied","Data":"3f2818e663f0f5e9ec32ed88de672b43618906c1c1860bb1f56673a0512fa8e0"} Jul 10 18:42:22.236774 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.236750 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:42:22.367347 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367260 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tls-certs\") pod \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " Jul 10 18:42:22.367347 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367304 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-dshm\") pod \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " Jul 10 18:42:22.367347 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367348 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnwn5\" (UniqueName: \"kubernetes.io/projected/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-kube-api-access-gnwn5\") pod \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " Jul 10 18:42:22.367762 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367377 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-home\") pod \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " Jul 10 18:42:22.367762 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367420 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-model-cache\") pod \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " Jul 10 18:42:22.367762 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367514 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tmp-dir\") pod \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\" (UID: \"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1\") " Jul 10 18:42:22.367762 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367705 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-home" (OuterVolumeSpecName: "home") pod "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" (UID: "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:22.367911 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367772 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" (UID: "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:22.367911 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367796 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" (UID: "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:22.367911 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367872 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tmp-dir\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:22.367911 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367890 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-home\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:22.367911 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.367903 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-model-cache\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:22.369738 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.369716 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-dshm" (OuterVolumeSpecName: "dshm") pod "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" (UID: "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:22.369738 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.369729 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" (UID: "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:42:22.369852 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.369809 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-kube-api-access-gnwn5" (OuterVolumeSpecName: "kube-api-access-gnwn5") pod "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" (UID: "e4cff85c-f38d-48d5-b2a7-c1940ceddcc1"). InnerVolumeSpecName "kube-api-access-gnwn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:42:22.468507 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.468468 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-tls-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:22.468507 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.468499 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-dshm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:22.468507 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:22.468512 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnwn5\" (UniqueName: \"kubernetes.io/projected/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1-kube-api-access-gnwn5\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:23.081697 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.081656 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" podUID="9b6de3b9-72f9-420b-bc65-c144e6db40b6" containerName="main" containerID="cri-o://f32f5a514ebcbfe4574cbd2dd2394142c19a02c084842227324c6707b99f252e" gracePeriod=45 Jul 10 18:42:23.202489 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.202459 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" Jul 10 18:42:23.202669 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.202486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl" event={"ID":"e4cff85c-f38d-48d5-b2a7-c1940ceddcc1","Type":"ContainerDied","Data":"5b0dbe10ba32c5be6f3ab8f9ff0068ce6f30e0137f9ce7de8980955a4ca56d47"} Jul 10 18:42:23.202669 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.202554 2570 scope.go:117] "RemoveContainer" containerID="3f2818e663f0f5e9ec32ed88de672b43618906c1c1860bb1f56673a0512fa8e0" Jul 10 18:42:23.204255 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.204231 2570 generic.go:356] "Generic (PLEG): container finished" podID="9b6de3b9-72f9-420b-bc65-c144e6db40b6" containerID="f32f5a514ebcbfe4574cbd2dd2394142c19a02c084842227324c6707b99f252e" exitCode=0 Jul 10 18:42:23.204385 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.204265 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" event={"ID":"9b6de3b9-72f9-420b-bc65-c144e6db40b6","Type":"ContainerDied","Data":"f32f5a514ebcbfe4574cbd2dd2394142c19a02c084842227324c6707b99f252e"} Jul 10 18:42:23.224765 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.224736 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl"] Jul 10 18:42:23.229714 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.229064 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/sub-test-model-319db5-kserve-7746758695-4lrtl"] Jul 10 18:42:23.337993 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.337935 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:42:23.463408 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.463374 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" path="/var/lib/kubelet/pods/e4cff85c-f38d-48d5-b2a7-c1940ceddcc1/volumes" Jul 10 18:42:23.477402 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477379 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-home\") pod \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " Jul 10 18:42:23.477555 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477420 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tls-certs\") pod \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " Jul 10 18:42:23.477555 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477450 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-dshm\") pod \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " Jul 10 18:42:23.477555 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477513 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-model-cache\") pod \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " Jul 10 18:42:23.477742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477571 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz2s6\" (UniqueName: \"kubernetes.io/projected/9b6de3b9-72f9-420b-bc65-c144e6db40b6-kube-api-access-hz2s6\") pod \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " Jul 10 18:42:23.477742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477673 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tmp-dir\") pod \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\" (UID: \"9b6de3b9-72f9-420b-bc65-c144e6db40b6\") " Jul 10 18:42:23.477742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477706 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-home" (OuterVolumeSpecName: "home") pod "9b6de3b9-72f9-420b-bc65-c144e6db40b6" (UID: "9b6de3b9-72f9-420b-bc65-c144e6db40b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:23.477894 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477811 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b6de3b9-72f9-420b-bc65-c144e6db40b6" (UID: "9b6de3b9-72f9-420b-bc65-c144e6db40b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:23.477964 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477948 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-home\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:23.478019 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477973 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-model-cache\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:23.478019 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.477942 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b6de3b9-72f9-420b-bc65-c144e6db40b6" (UID: "9b6de3b9-72f9-420b-bc65-c144e6db40b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:23.479725 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.479695 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b6de3b9-72f9-420b-bc65-c144e6db40b6" (UID: "9b6de3b9-72f9-420b-bc65-c144e6db40b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:42:23.479841 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.479741 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-dshm" (OuterVolumeSpecName: "dshm") pod "9b6de3b9-72f9-420b-bc65-c144e6db40b6" (UID: "9b6de3b9-72f9-420b-bc65-c144e6db40b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:42:23.479841 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.479795 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6de3b9-72f9-420b-bc65-c144e6db40b6-kube-api-access-hz2s6" (OuterVolumeSpecName: "kube-api-access-hz2s6") pod "9b6de3b9-72f9-420b-bc65-c144e6db40b6" (UID: "9b6de3b9-72f9-420b-bc65-c144e6db40b6"). InnerVolumeSpecName "kube-api-access-hz2s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:42:23.578394 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.578347 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tmp-dir\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:23.578394 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.578386 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b6de3b9-72f9-420b-bc65-c144e6db40b6-tls-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:23.578394 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.578395 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b6de3b9-72f9-420b-bc65-c144e6db40b6-dshm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:23.578394 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:23.578404 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz2s6\" (UniqueName: \"kubernetes.io/projected/9b6de3b9-72f9-420b-bc65-c144e6db40b6-kube-api-access-hz2s6\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:42:24.210939 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:24.210907 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" Jul 10 18:42:24.211380 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:24.210945 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj" event={"ID":"9b6de3b9-72f9-420b-bc65-c144e6db40b6","Type":"ContainerDied","Data":"a9b96f1c1bc6558ca8cc055bd24bd92ff990b362e9b9156c664a0f2372b086ff"} Jul 10 18:42:24.211380 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:24.210996 2570 scope.go:117] "RemoveContainer" containerID="f32f5a514ebcbfe4574cbd2dd2394142c19a02c084842227324c6707b99f252e" Jul 10 18:42:24.232135 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:24.232111 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj"] Jul 10 18:42:24.238259 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:24.238233 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/sub-test-model-4cfdb5-kserve-ddb77bcf-fsnlj"] Jul 10 18:42:25.463815 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:25.463770 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b6de3b9-72f9-420b-bc65-c144e6db40b6" path="/var/lib/kubelet/pods/9b6de3b9-72f9-420b-bc65-c144e6db40b6/volumes" Jul 10 18:42:38.496868 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.496833 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm"] Jul 10 18:42:38.497237 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.497157 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6de3b9-72f9-420b-bc65-c144e6db40b6" containerName="main" Jul 10 18:42:38.497237 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.497167 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6de3b9-72f9-420b-bc65-c144e6db40b6" containerName="main" Jul 10 18:42:38.497237 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.497177 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" containerName="main" Jul 10 18:42:38.497237 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.497182 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" containerName="main" Jul 10 18:42:38.497374 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.497239 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cff85c-f38d-48d5-b2a7-c1940ceddcc1" containerName="main" Jul 10 18:42:38.497374 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.497249 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6de3b9-72f9-420b-bc65-c144e6db40b6" containerName="main" Jul 10 18:42:38.501335 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.501314 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.505776 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.505755 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"rate-test-model-319db5-kserve-self-signed-certs\"" Jul 10 18:42:38.505909 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.505791 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"openshift-service-ca.crt\"" Jul 10 18:42:38.505909 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.505757 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"kube-root-ca.crt\"" Jul 10 18:42:38.505909 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.505757 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-319db5\"/\"default-dockercfg-s8p4h\"" Jul 10 18:42:38.513999 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.513977 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-home\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.514106 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.514029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-tmp-dir\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.514106 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.514067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7x8p\" (UniqueName: \"kubernetes.io/projected/17712ab5-46a9-480e-b201-74691c28b5a0-kube-api-access-w7x8p\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.514210 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.514135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-dshm\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.514210 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.514170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-model-cache\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.514210 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.514196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17712ab5-46a9-480e-b201-74691c28b5a0-tls-certs\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.519726 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.519706 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm"] Jul 10 18:42:38.614779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.614743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-dshm\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.614779 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.614787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-model-cache\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.615051 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.614810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17712ab5-46a9-480e-b201-74691c28b5a0-tls-certs\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.615051 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.614836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-home\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.615051 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.614867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-tmp-dir\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.615051 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.614886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7x8p\" (UniqueName: \"kubernetes.io/projected/17712ab5-46a9-480e-b201-74691c28b5a0-kube-api-access-w7x8p\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.615301 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.615279 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-model-cache\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.615366 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.615317 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-tmp-dir\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.615366 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.615344 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-home\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.617243 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.617219 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-dshm\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.617519 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.617500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17712ab5-46a9-480e-b201-74691c28b5a0-tls-certs\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.623238 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.623215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7x8p\" (UniqueName: \"kubernetes.io/projected/17712ab5-46a9-480e-b201-74691c28b5a0-kube-api-access-w7x8p\") pod \"rate-test-model-319db5-kserve-59bd95fc67-x2vwm\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.810517 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.810426 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:38.947911 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:38.947883 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm"] Jul 10 18:42:38.949936 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:42:38.949910 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17712ab5_46a9_480e_b201_74691c28b5a0.slice/crio-0fc65e8a68fee3cc1727d12c982002d9fec975c239af99fab5a12fb602d82750 WatchSource:0}: Error finding container 0fc65e8a68fee3cc1727d12c982002d9fec975c239af99fab5a12fb602d82750: Status 404 returned error can't find the container with id 0fc65e8a68fee3cc1727d12c982002d9fec975c239af99fab5a12fb602d82750 Jul 10 18:42:39.287634 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:39.287587 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" event={"ID":"17712ab5-46a9-480e-b201-74691c28b5a0","Type":"ContainerStarted","Data":"bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33"} Jul 10 18:42:39.287812 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:39.287648 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" event={"ID":"17712ab5-46a9-480e-b201-74691c28b5a0","Type":"ContainerStarted","Data":"0fc65e8a68fee3cc1727d12c982002d9fec975c239af99fab5a12fb602d82750"} Jul 10 18:42:39.305238 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:39.305188 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" podStartSLOduration=1.305171958 podStartE2EDuration="1.305171958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:42:39.304418603 +0000 UTC m=+2368.399609088" watchObservedRunningTime="2026-07-10 18:42:39.305171958 +0000 UTC m=+2368.400362445" Jul 10 18:42:48.810917 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:48.810877 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:48.811311 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:48.810937 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:48.823846 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:48.823823 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:49.356504 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:49.356477 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:42:59.627049 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.627014 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb"] Jul 10 18:42:59.630376 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.630359 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.633326 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.633298 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"kube-root-ca.crt\"" Jul 10 18:42:59.634365 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.634328 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"default-dockercfg-mf8f5\"" Jul 10 18:42:59.634498 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.634360 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"openshift-service-ca.crt\"" Jul 10 18:42:59.634498 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.634417 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4cfdb5\"/\"rate-test-model-4cfdb5-kserve-self-signed-certs\"" Jul 10 18:42:59.651272 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.651246 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb"] Jul 10 18:42:59.691440 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.691412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2286a96-5f54-4281-a55b-4779bb83cc32-tls-certs\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.691592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.691445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-tmp-dir\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.691592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.691478 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-dshm\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.691592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.691528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-model-cache\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.691592 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.691556 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplv6\" (UniqueName: \"kubernetes.io/projected/c2286a96-5f54-4281-a55b-4779bb83cc32-kube-api-access-wplv6\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.691761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.691637 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-home\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792344 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-dshm\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792344 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792345 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-model-cache\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792567 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplv6\" (UniqueName: \"kubernetes.io/projected/c2286a96-5f54-4281-a55b-4779bb83cc32-kube-api-access-wplv6\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792567 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-home\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792701 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2286a96-5f54-4281-a55b-4779bb83cc32-tls-certs\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792701 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792665 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-tmp-dir\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792889 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792866 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-model-cache\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792962 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792877 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-home\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.792962 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.792923 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-tmp-dir\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.794909 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.794887 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-dshm\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.795195 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.795176 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2286a96-5f54-4281-a55b-4779bb83cc32-tls-certs\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.805937 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.805910 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplv6\" (UniqueName: \"kubernetes.io/projected/c2286a96-5f54-4281-a55b-4779bb83cc32-kube-api-access-wplv6\") pod \"rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:42:59.940394 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:42:59.940356 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:43:00.088350 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:00.088322 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb"] Jul 10 18:43:00.090185 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:43:00.090154 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2286a96_5f54_4281_a55b_4779bb83cc32.slice/crio-715224014af6fa2db7512d9ff4405661468802984dcf8cba00465ca8ed3d1355 WatchSource:0}: Error finding container 715224014af6fa2db7512d9ff4405661468802984dcf8cba00465ca8ed3d1355: Status 404 returned error can't find the container with id 715224014af6fa2db7512d9ff4405661468802984dcf8cba00465ca8ed3d1355 Jul 10 18:43:00.402366 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:00.402272 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" event={"ID":"c2286a96-5f54-4281-a55b-4779bb83cc32","Type":"ContainerStarted","Data":"12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb"} Jul 10 18:43:00.402366 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:00.402316 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" event={"ID":"c2286a96-5f54-4281-a55b-4779bb83cc32","Type":"ContainerStarted","Data":"715224014af6fa2db7512d9ff4405661468802984dcf8cba00465ca8ed3d1355"} Jul 10 18:43:00.439245 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:00.439192 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" podStartSLOduration=1.439162288 podStartE2EDuration="1.439162288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:43:00.437774299 +0000 UTC m=+2389.532964785" watchObservedRunningTime="2026-07-10 18:43:00.439162288 +0000 UTC m=+2389.534352830" Jul 10 18:43:09.940932 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:09.940894 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:43:09.941451 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:09.941191 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:43:09.953971 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:09.953947 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:43:10.462441 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:10.462408 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:43:11.575911 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:11.575803 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:43:11.588209 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:11.588188 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:43:51.419888 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:51.419848 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm"] Jul 10 18:43:52.405658 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:43:52.405621 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb"] Jul 10 18:44:06.440796 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.440721 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" podUID="17712ab5-46a9-480e-b201-74691c28b5a0" containerName="main" containerID="cri-o://bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33" gracePeriod=45 Jul 10 18:44:06.690587 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.690559 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:44:06.750071 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.749982 2570 generic.go:356] "Generic (PLEG): container finished" podID="17712ab5-46a9-480e-b201-74691c28b5a0" containerID="bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33" exitCode=0 Jul 10 18:44:06.750071 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.750023 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" Jul 10 18:44:06.750071 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.750049 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" event={"ID":"17712ab5-46a9-480e-b201-74691c28b5a0","Type":"ContainerDied","Data":"bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33"} Jul 10 18:44:06.750304 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.750081 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm" event={"ID":"17712ab5-46a9-480e-b201-74691c28b5a0","Type":"ContainerDied","Data":"0fc65e8a68fee3cc1727d12c982002d9fec975c239af99fab5a12fb602d82750"} Jul 10 18:44:06.750304 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.750099 2570 scope.go:117] "RemoveContainer" containerID="bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33" Jul 10 18:44:06.759000 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.758979 2570 scope.go:117] "RemoveContainer" containerID="bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33" Jul 10 18:44:06.759307 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:44:06.759277 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33\": container with ID starting with bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33 not found: ID does not exist" containerID="bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33" Jul 10 18:44:06.759368 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.759316 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33"} err="failed to get container status \"bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33\": rpc error: code = NotFound desc = could not find container \"bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33\": container with ID starting with bd9ea63be21f3c3e4b708ab70b4e3fa5ce672798d772003cc9ed2da700f49c33 not found: ID does not exist" Jul 10 18:44:06.812068 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812029 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7x8p\" (UniqueName: \"kubernetes.io/projected/17712ab5-46a9-480e-b201-74691c28b5a0-kube-api-access-w7x8p\") pod \"17712ab5-46a9-480e-b201-74691c28b5a0\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " Jul 10 18:44:06.812068 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812071 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17712ab5-46a9-480e-b201-74691c28b5a0-tls-certs\") pod \"17712ab5-46a9-480e-b201-74691c28b5a0\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " Jul 10 18:44:06.812333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812118 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-tmp-dir\") pod \"17712ab5-46a9-480e-b201-74691c28b5a0\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " Jul 10 18:44:06.812333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812149 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-home\") pod \"17712ab5-46a9-480e-b201-74691c28b5a0\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " Jul 10 18:44:06.812333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812175 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-model-cache\") pod \"17712ab5-46a9-480e-b201-74691c28b5a0\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " Jul 10 18:44:06.812333 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812193 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-dshm\") pod \"17712ab5-46a9-480e-b201-74691c28b5a0\" (UID: \"17712ab5-46a9-480e-b201-74691c28b5a0\") " Jul 10 18:44:06.812534 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812416 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-home" (OuterVolumeSpecName: "home") pod "17712ab5-46a9-480e-b201-74691c28b5a0" (UID: "17712ab5-46a9-480e-b201-74691c28b5a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:06.812534 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812430 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17712ab5-46a9-480e-b201-74691c28b5a0" (UID: "17712ab5-46a9-480e-b201-74691c28b5a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:06.812534 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.812491 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "17712ab5-46a9-480e-b201-74691c28b5a0" (UID: "17712ab5-46a9-480e-b201-74691c28b5a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:06.814526 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.814504 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17712ab5-46a9-480e-b201-74691c28b5a0-kube-api-access-w7x8p" (OuterVolumeSpecName: "kube-api-access-w7x8p") pod "17712ab5-46a9-480e-b201-74691c28b5a0" (UID: "17712ab5-46a9-480e-b201-74691c28b5a0"). InnerVolumeSpecName "kube-api-access-w7x8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:44:06.814635 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.814546 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17712ab5-46a9-480e-b201-74691c28b5a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17712ab5-46a9-480e-b201-74691c28b5a0" (UID: "17712ab5-46a9-480e-b201-74691c28b5a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:44:06.814635 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.814580 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-dshm" (OuterVolumeSpecName: "dshm") pod "17712ab5-46a9-480e-b201-74691c28b5a0" (UID: "17712ab5-46a9-480e-b201-74691c28b5a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:06.913867 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.913829 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-tmp-dir\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:06.913867 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.913860 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-home\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:06.913867 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.913870 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-model-cache\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:06.914105 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.913878 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17712ab5-46a9-480e-b201-74691c28b5a0-dshm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:06.914105 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.913887 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7x8p\" (UniqueName: \"kubernetes.io/projected/17712ab5-46a9-480e-b201-74691c28b5a0-kube-api-access-w7x8p\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:06.914105 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:06.913896 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17712ab5-46a9-480e-b201-74691c28b5a0-tls-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:07.075121 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.075078 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm"] Jul 10 18:44:07.078719 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.078687 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-319db5/rate-test-model-319db5-kserve-59bd95fc67-x2vwm"] Jul 10 18:44:07.426959 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.426897 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" podUID="c2286a96-5f54-4281-a55b-4779bb83cc32" containerName="main" containerID="cri-o://12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb" gracePeriod=45 Jul 10 18:44:07.464349 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.464314 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17712ab5-46a9-480e-b201-74691c28b5a0" path="/var/lib/kubelet/pods/17712ab5-46a9-480e-b201-74691c28b5a0/volumes" Jul 10 18:44:07.691195 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.691115 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:44:07.762782 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.762737 2570 generic.go:356] "Generic (PLEG): container finished" podID="c2286a96-5f54-4281-a55b-4779bb83cc32" containerID="12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb" exitCode=0 Jul 10 18:44:07.762992 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.762803 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" Jul 10 18:44:07.762992 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.762821 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" event={"ID":"c2286a96-5f54-4281-a55b-4779bb83cc32","Type":"ContainerDied","Data":"12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb"} Jul 10 18:44:07.762992 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.762870 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb" event={"ID":"c2286a96-5f54-4281-a55b-4779bb83cc32","Type":"ContainerDied","Data":"715224014af6fa2db7512d9ff4405661468802984dcf8cba00465ca8ed3d1355"} Jul 10 18:44:07.762992 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.762889 2570 scope.go:117] "RemoveContainer" containerID="12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb" Jul 10 18:44:07.774278 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.774259 2570 scope.go:117] "RemoveContainer" containerID="12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb" Jul 10 18:44:07.774545 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:44:07.774524 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb\": container with ID starting with 12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb not found: ID does not exist" containerID="12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb" Jul 10 18:44:07.774619 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.774556 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb"} err="failed to get container status \"12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb\": rpc error: code = NotFound desc = could not find container \"12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb\": container with ID starting with 12059e0b99d1a452081b04a450a3c206d966480dccd2358f1057ba918a2ed7cb not found: ID does not exist" Jul 10 18:44:07.822540 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822502 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2286a96-5f54-4281-a55b-4779bb83cc32-tls-certs\") pod \"c2286a96-5f54-4281-a55b-4779bb83cc32\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " Jul 10 18:44:07.822540 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822543 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-model-cache\") pod \"c2286a96-5f54-4281-a55b-4779bb83cc32\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " Jul 10 18:44:07.822818 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822584 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wplv6\" (UniqueName: \"kubernetes.io/projected/c2286a96-5f54-4281-a55b-4779bb83cc32-kube-api-access-wplv6\") pod \"c2286a96-5f54-4281-a55b-4779bb83cc32\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " Jul 10 18:44:07.822818 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822624 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-home\") pod \"c2286a96-5f54-4281-a55b-4779bb83cc32\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " Jul 10 18:44:07.822818 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822726 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-dshm\") pod \"c2286a96-5f54-4281-a55b-4779bb83cc32\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " Jul 10 18:44:07.822818 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822748 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-tmp-dir\") pod \"c2286a96-5f54-4281-a55b-4779bb83cc32\" (UID: \"c2286a96-5f54-4281-a55b-4779bb83cc32\") " Jul 10 18:44:07.823025 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822891 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2286a96-5f54-4281-a55b-4779bb83cc32" (UID: "c2286a96-5f54-4281-a55b-4779bb83cc32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:07.823025 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.822961 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-home" (OuterVolumeSpecName: "home") pod "c2286a96-5f54-4281-a55b-4779bb83cc32" (UID: "c2286a96-5f54-4281-a55b-4779bb83cc32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:07.823108 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.823051 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-model-cache\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:07.823108 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.823071 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-home\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:07.823179 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.823112 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2286a96-5f54-4281-a55b-4779bb83cc32" (UID: "c2286a96-5f54-4281-a55b-4779bb83cc32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:07.824994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.824965 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2286a96-5f54-4281-a55b-4779bb83cc32-kube-api-access-wplv6" (OuterVolumeSpecName: "kube-api-access-wplv6") pod "c2286a96-5f54-4281-a55b-4779bb83cc32" (UID: "c2286a96-5f54-4281-a55b-4779bb83cc32"). InnerVolumeSpecName "kube-api-access-wplv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:44:07.824994 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.824973 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-dshm" (OuterVolumeSpecName: "dshm") pod "c2286a96-5f54-4281-a55b-4779bb83cc32" (UID: "c2286a96-5f54-4281-a55b-4779bb83cc32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:44:07.825148 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.825008 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2286a96-5f54-4281-a55b-4779bb83cc32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2286a96-5f54-4281-a55b-4779bb83cc32" (UID: "c2286a96-5f54-4281-a55b-4779bb83cc32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:44:07.923952 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.923914 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2286a96-5f54-4281-a55b-4779bb83cc32-tls-certs\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:07.923952 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.923943 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplv6\" (UniqueName: \"kubernetes.io/projected/c2286a96-5f54-4281-a55b-4779bb83cc32-kube-api-access-wplv6\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:07.923952 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.923954 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-dshm\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:07.924187 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:07.923986 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2286a96-5f54-4281-a55b-4779bb83cc32-tmp-dir\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:08.085894 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.085858 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb"] Jul 10 18:44:08.092414 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.092385 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4cfdb5/rate-test-model-4cfdb5-kserve-664cc489f5-kjcdb"] Jul 10 18:44:08.957322 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.957280 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7380-139772321405808"] Jul 10 18:44:08.957742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.957651 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17712ab5-46a9-480e-b201-74691c28b5a0" containerName="main" Jul 10 18:44:08.957742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.957665 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="17712ab5-46a9-480e-b201-74691c28b5a0" containerName="main" Jul 10 18:44:08.957742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.957682 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2286a96-5f54-4281-a55b-4779bb83cc32" containerName="main" Jul 10 18:44:08.957742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.957687 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2286a96-5f54-4281-a55b-4779bb83cc32" containerName="main" Jul 10 18:44:08.957873 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.957745 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="17712ab5-46a9-480e-b201-74691c28b5a0" containerName="main" Jul 10 18:44:08.957873 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.957754 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2286a96-5f54-4281-a55b-4779bb83cc32" containerName="main" Jul 10 18:44:08.961964 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.961943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:08.965070 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.965046 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-kjqpt\"" Jul 10 18:44:08.966770 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:08.966749 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7380-139772321405808"] Jul 10 18:44:09.135484 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.135444 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glxd8\" (UniqueName: \"kubernetes.io/projected/0f41b930-0744-4b73-a528-835de5f45488-kube-api-access-glxd8\") pod \"test-curl-7380-139772321405808\" (UID: \"0f41b930-0744-4b73-a528-835de5f45488\") " pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:09.236810 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.236711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glxd8\" (UniqueName: \"kubernetes.io/projected/0f41b930-0744-4b73-a528-835de5f45488-kube-api-access-glxd8\") pod \"test-curl-7380-139772321405808\" (UID: \"0f41b930-0744-4b73-a528-835de5f45488\") " pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:09.245899 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.245864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glxd8\" (UniqueName: \"kubernetes.io/projected/0f41b930-0744-4b73-a528-835de5f45488-kube-api-access-glxd8\") pod \"test-curl-7380-139772321405808\" (UID: \"0f41b930-0744-4b73-a528-835de5f45488\") " pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:09.272407 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.272363 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:09.410665 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.410631 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7380-139772321405808"] Jul 10 18:44:09.412135 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:44:09.412106 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f41b930_0744_4b73_a528_835de5f45488.slice/crio-2c40af306811cfa2e311a332dccd532edf30ea201cd32477908369c366f94620 WatchSource:0}: Error finding container 2c40af306811cfa2e311a332dccd532edf30ea201cd32477908369c366f94620: Status 404 returned error can't find the container with id 2c40af306811cfa2e311a332dccd532edf30ea201cd32477908369c366f94620 Jul 10 18:44:09.413365 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.413349 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:44:09.463894 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.463865 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2286a96-5f54-4281-a55b-4779bb83cc32" path="/var/lib/kubelet/pods/c2286a96-5f54-4281-a55b-4779bb83cc32/volumes" Jul 10 18:44:09.777670 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:09.777623 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7380-139772321405808" event={"ID":"0f41b930-0744-4b73-a528-835de5f45488","Type":"ContainerStarted","Data":"2c40af306811cfa2e311a332dccd532edf30ea201cd32477908369c366f94620"} Jul 10 18:44:10.783970 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:10.783936 2570 generic.go:356] "Generic (PLEG): container finished" podID="0f41b930-0744-4b73-a528-835de5f45488" containerID="b593246d2c524b8adb71ba3621b275ab0805e914692965341fd9dd6ee894c864" exitCode=0 Jul 10 18:44:10.784359 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:10.784024 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7380-139772321405808" event={"ID":"0f41b930-0744-4b73-a528-835de5f45488","Type":"ContainerDied","Data":"b593246d2c524b8adb71ba3621b275ab0805e914692965341fd9dd6ee894c864"} Jul 10 18:44:11.934985 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:11.934959 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:11.971547 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:11.971521 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7380-139772321405808_0f41b930-0744-4b73-a528-835de5f45488/test-curl-7380-139772321405808/0.log" Jul 10 18:44:11.996248 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:11.996215 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7380-139772321405808"] Jul 10 18:44:12.000167 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:12.000141 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7380-139772321405808"] Jul 10 18:44:12.061109 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:12.061081 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glxd8\" (UniqueName: \"kubernetes.io/projected/0f41b930-0744-4b73-a528-835de5f45488-kube-api-access-glxd8\") pod \"0f41b930-0744-4b73-a528-835de5f45488\" (UID: \"0f41b930-0744-4b73-a528-835de5f45488\") " Jul 10 18:44:12.063415 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:12.063381 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f41b930-0744-4b73-a528-835de5f45488-kube-api-access-glxd8" (OuterVolumeSpecName: "kube-api-access-glxd8") pod "0f41b930-0744-4b73-a528-835de5f45488" (UID: "0f41b930-0744-4b73-a528-835de5f45488"). InnerVolumeSpecName "kube-api-access-glxd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:44:12.162576 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:12.162543 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glxd8\" (UniqueName: \"kubernetes.io/projected/0f41b930-0744-4b73-a528-835de5f45488-kube-api-access-glxd8\") on node \"ip-10-0-141-217.ec2.internal\" DevicePath \"\"" Jul 10 18:44:12.796095 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:12.796062 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c40af306811cfa2e311a332dccd532edf30ea201cd32477908369c366f94620" Jul 10 18:44:12.796095 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:12.796097 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:12.808314 ip-10-0-141-217 kubenswrapper[2570]: E0710 18:44:12.808269 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7380-139772321405808\" is forbidden: User \"system:node:ip-10-0-141-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-217.ec2.internal' and this object" podUID="0f41b930-0744-4b73-a528-835de5f45488" pod="openshift-ingress/test-curl-7380-139772321405808" Jul 10 18:44:13.464162 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:13.464119 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f41b930-0744-4b73-a528-835de5f45488" path="/var/lib/kubelet/pods/0f41b930-0744-4b73-a528-835de5f45488/volumes" Jul 10 18:44:26.828346 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:26.828312 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6789f574f6-mzvs9_75cc925b-40bd-443f-a84e-45d2ed9bcd17/authorino/0.log" Jul 10 18:44:31.794705 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:31.794669 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4jb82_1a43527f-276f-4d39-bf1d-3146106459a6/server/0.log" Jul 10 18:44:31.924818 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:31.924780 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ps68r_a3762dab-7ad1-440e-bbff-d8a6df59eccf/manager/0.log" Jul 10 18:44:32.050551 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:32.050478 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795b486df7-4bvw8_52a7d721-1131-4935-909b-043dd4f03e57/copy-manifests/0.log" Jul 10 18:44:32.069184 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:32.069148 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795b486df7-4bvw8_52a7d721-1131-4935-909b-043dd4f03e57/manager/0.log" Jul 10 18:44:33.485193 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.485149 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p_8455b78d-7899-48ee-a4c4-f694f3b04396/util/0.log" Jul 10 18:44:33.494796 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.494769 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p_8455b78d-7899-48ee-a4c4-f694f3b04396/pull/0.log" Jul 10 18:44:33.503146 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.503122 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p_8455b78d-7899-48ee-a4c4-f694f3b04396/extract/0.log" Jul 10 18:44:33.614574 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.614544 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9_a7adf02a-dabf-45ac-b0f3-35e1b5791aae/util/0.log" Jul 10 18:44:33.621999 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.621978 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9_a7adf02a-dabf-45ac-b0f3-35e1b5791aae/pull/0.log" Jul 10 18:44:33.629210 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.629190 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9_a7adf02a-dabf-45ac-b0f3-35e1b5791aae/extract/0.log" Jul 10 18:44:33.732962 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.732931 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk_b790e7e2-11fa-4cba-9acf-a38c72615195/util/0.log" Jul 10 18:44:33.739783 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.739714 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk_b790e7e2-11fa-4cba-9acf-a38c72615195/pull/0.log" Jul 10 18:44:33.746348 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.746330 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk_b790e7e2-11fa-4cba-9acf-a38c72615195/extract/0.log" Jul 10 18:44:33.854224 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.854194 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9_c2b5eebb-cae4-4902-af89-79b06ed4b7f7/util/0.log" Jul 10 18:44:33.861128 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.861102 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9_c2b5eebb-cae4-4902-af89-79b06ed4b7f7/pull/0.log" Jul 10 18:44:33.868798 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.868774 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9_c2b5eebb-cae4-4902-af89-79b06ed4b7f7/extract/0.log" Jul 10 18:44:33.989149 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:33.989118 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6789f574f6-mzvs9_75cc925b-40bd-443f-a84e-45d2ed9bcd17/authorino/0.log" Jul 10 18:44:34.319188 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:34.319155 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-x7z2r_36aab6cf-13c3-4a26-9127-de4e8f67cd31/kuadrant-console-plugin/0.log" Jul 10 18:44:35.453938 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:35.453907 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-655946c88b-vf5p4_dd0ff8f4-884b-4102-bd8a-409bad35e7b6/kube-auth-proxy/0.log" Jul 10 18:44:35.684009 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:35.683978 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-gmms5_6c08f902-e491-4458-ab6c-aade2b0e69d1/payload-pre-processing/0.log" Jul 10 18:44:36.239057 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:36.239019 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-g2jzh_f0ce4e45-4bea-4309-824d-17a4d65f93c5/main/0.log" Jul 10 18:44:36.562479 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:36.562398 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm5hr7_fbb85468-134c-4f00-a735-b2f4e838dff0/main/0.log" Jul 10 18:44:36.783692 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:36.783663 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-vjzms_db947848-2de9-4eae-a073-9c007350e923/main/0.log" Jul 10 18:44:40.977283 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.977243 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rjmm9/must-gather-pmhcc"] Jul 10 18:44:40.977698 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.977565 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f41b930-0744-4b73-a528-835de5f45488" containerName="test-curl-7380-139772321405808" Jul 10 18:44:40.977698 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.977577 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f41b930-0744-4b73-a528-835de5f45488" containerName="test-curl-7380-139772321405808" Jul 10 18:44:40.977698 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.977655 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f41b930-0744-4b73-a528-835de5f45488" containerName="test-curl-7380-139772321405808" Jul 10 18:44:40.980741 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.980720 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:40.983757 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.983740 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rjmm9\"/\"default-dockercfg-r9c95\"" Jul 10 18:44:40.984465 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.984446 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rjmm9\"/\"kube-root-ca.crt\"" Jul 10 18:44:40.984522 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.984512 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rjmm9\"/\"openshift-service-ca.crt\"" Jul 10 18:44:40.991143 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:40.991121 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rjmm9/must-gather-pmhcc"] Jul 10 18:44:41.099340 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.099306 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdjx\" (UniqueName: \"kubernetes.io/projected/b8112c49-4ead-4aa6-821e-d31e8437dfda-kube-api-access-5xdjx\") pod \"must-gather-pmhcc\" (UID: \"b8112c49-4ead-4aa6-821e-d31e8437dfda\") " pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:41.099340 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.099345 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b8112c49-4ead-4aa6-821e-d31e8437dfda-must-gather-output\") pod \"must-gather-pmhcc\" (UID: \"b8112c49-4ead-4aa6-821e-d31e8437dfda\") " pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:41.199802 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.199764 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdjx\" (UniqueName: \"kubernetes.io/projected/b8112c49-4ead-4aa6-821e-d31e8437dfda-kube-api-access-5xdjx\") pod \"must-gather-pmhcc\" (UID: \"b8112c49-4ead-4aa6-821e-d31e8437dfda\") " pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:41.199802 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.199808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b8112c49-4ead-4aa6-821e-d31e8437dfda-must-gather-output\") pod \"must-gather-pmhcc\" (UID: \"b8112c49-4ead-4aa6-821e-d31e8437dfda\") " pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:41.200188 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.200165 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b8112c49-4ead-4aa6-821e-d31e8437dfda-must-gather-output\") pod \"must-gather-pmhcc\" (UID: \"b8112c49-4ead-4aa6-821e-d31e8437dfda\") " pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:41.211072 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.211041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdjx\" (UniqueName: \"kubernetes.io/projected/b8112c49-4ead-4aa6-821e-d31e8437dfda-kube-api-access-5xdjx\") pod \"must-gather-pmhcc\" (UID: \"b8112c49-4ead-4aa6-821e-d31e8437dfda\") " pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:41.290484 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.290405 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rjmm9/must-gather-pmhcc" Jul 10 18:44:41.425090 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.425049 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rjmm9/must-gather-pmhcc"] Jul 10 18:44:41.428592 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:44:41.428557 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8112c49_4ead_4aa6_821e_d31e8437dfda.slice/crio-485e593c0c8c999fc564948499c914fbf87faa05032c61bd7f0b69e48da976cc WatchSource:0}: Error finding container 485e593c0c8c999fc564948499c914fbf87faa05032c61bd7f0b69e48da976cc: Status 404 returned error can't find the container with id 485e593c0c8c999fc564948499c914fbf87faa05032c61bd7f0b69e48da976cc Jul 10 18:44:41.936478 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:41.936442 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjmm9/must-gather-pmhcc" event={"ID":"b8112c49-4ead-4aa6-821e-d31e8437dfda","Type":"ContainerStarted","Data":"485e593c0c8c999fc564948499c914fbf87faa05032c61bd7f0b69e48da976cc"} Jul 10 18:44:42.946132 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:42.946094 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjmm9/must-gather-pmhcc" event={"ID":"b8112c49-4ead-4aa6-821e-d31e8437dfda","Type":"ContainerStarted","Data":"178e0045a4792b452209dbe8bf50e1a57ce3ce8ea38bf708bed4f52664ea310f"} Jul 10 18:44:42.946516 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:42.946150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjmm9/must-gather-pmhcc" event={"ID":"b8112c49-4ead-4aa6-821e-d31e8437dfda","Type":"ContainerStarted","Data":"549c647dcab8c98dd421c15b62e814af5b293402a6c95d75054c320d6426cd46"} Jul 10 18:44:42.965086 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:42.965021 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rjmm9/must-gather-pmhcc" podStartSLOduration=2.137333334 podStartE2EDuration="2.964998048s" totalImagesPullingTime="827.664714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:44:42.961044095 +0000 UTC m=+2492.056234584" watchObservedRunningTime="2026-07-10 18:44:42.964998048 +0000 UTC m=+2492.060188534" Jul 10 18:44:43.810852 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:43.810816 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cl6p6_ce68a2f5-08f1-47cf-80c8-004a8ac733bb/global-pull-secret-syncer/0.log" Jul 10 18:44:44.046280 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:44.046242 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xp5fp_7d407443-3c8b-4af7-8e0e-537bc17cdf8e/konnectivity-agent/0.log" Jul 10 18:44:44.122198 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:44.122101 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-217.ec2.internal_fb22374ceca0abc340a036c543248c78/haproxy/0.log" Jul 10 18:44:48.085565 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.085524 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p_8455b78d-7899-48ee-a4c4-f694f3b04396/extract/0.log" Jul 10 18:44:48.121785 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.121758 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p_8455b78d-7899-48ee-a4c4-f694f3b04396/util/0.log" Jul 10 18:44:48.148705 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.148676 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599c65p_8455b78d-7899-48ee-a4c4-f694f3b04396/pull/0.log" Jul 10 18:44:48.178865 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.178827 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9_a7adf02a-dabf-45ac-b0f3-35e1b5791aae/extract/0.log" Jul 10 18:44:48.211259 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.211229 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9_a7adf02a-dabf-45ac-b0f3-35e1b5791aae/util/0.log" Jul 10 18:44:48.237151 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.237116 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02nxx9_a7adf02a-dabf-45ac-b0f3-35e1b5791aae/pull/0.log" Jul 10 18:44:48.269221 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.269190 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk_b790e7e2-11fa-4cba-9acf-a38c72615195/extract/0.log" Jul 10 18:44:48.294943 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.294914 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk_b790e7e2-11fa-4cba-9acf-a38c72615195/util/0.log" Jul 10 18:44:48.320958 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.320931 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v75wk_b790e7e2-11fa-4cba-9acf-a38c72615195/pull/0.log" Jul 10 18:44:48.356325 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.356224 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9_c2b5eebb-cae4-4902-af89-79b06ed4b7f7/extract/0.log" Jul 10 18:44:48.384488 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.384464 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9_c2b5eebb-cae4-4902-af89-79b06ed4b7f7/util/0.log" Jul 10 18:44:48.414003 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.413967 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef188dc9_c2b5eebb-cae4-4902-af89-79b06ed4b7f7/pull/0.log" Jul 10 18:44:48.863482 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.863442 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6789f574f6-mzvs9_75cc925b-40bd-443f-a84e-45d2ed9bcd17/authorino/0.log" Jul 10 18:44:48.974761 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:48.974719 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-x7z2r_36aab6cf-13c3-4a26-9127-de4e8f67cd31/kuadrant-console-plugin/0.log" Jul 10 18:44:51.114211 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.114132 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-689b6dc5d-rnw2l_3df3abe4-98ff-4030-ba73-7a3938448563/metrics-server/0.log" Jul 10 18:44:51.178245 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.178210 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8686k_14c66d29-b215-466b-a6b7-cd0451b307bb/node-exporter/0.log" Jul 10 18:44:51.200094 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.200055 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8686k_14c66d29-b215-466b-a6b7-cd0451b307bb/kube-rbac-proxy/0.log" Jul 10 18:44:51.222385 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.222351 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8686k_14c66d29-b215-466b-a6b7-cd0451b307bb/init-textfile/0.log" Jul 10 18:44:51.811825 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.811782 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-vk96k_7af12151-b236-461f-a999-7615f8bfbdbe/prometheus-operator-admission-webhook/0.log" Jul 10 18:44:51.846119 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.846068 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dbfc976f4-q8wzr_511ef8f2-df2d-473a-87b7-d61081033910/telemeter-client/0.log" Jul 10 18:44:51.870618 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.870559 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dbfc976f4-q8wzr_511ef8f2-df2d-473a-87b7-d61081033910/reload/0.log" Jul 10 18:44:51.897093 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:51.897058 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dbfc976f4-q8wzr_511ef8f2-df2d-473a-87b7-d61081033910/kube-rbac-proxy/0.log" Jul 10 18:44:54.457760 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.457703 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d"] Jul 10 18:44:54.466371 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.466334 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.468727 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.468695 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d"] Jul 10 18:44:54.538396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.538072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-lib-modules\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.538396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.538152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnz9s\" (UniqueName: \"kubernetes.io/projected/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-kube-api-access-xnz9s\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.538396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.538190 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-podres\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.538396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.538221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-sys\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.538396 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.538282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-proc\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639319 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-proc\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-lib-modules\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnz9s\" (UniqueName: \"kubernetes.io/projected/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-kube-api-access-xnz9s\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639428 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-podres\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639438 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-proc\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-sys\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639509 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-sys\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639890 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-podres\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.639890 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.639547 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-lib-modules\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.649048 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.649016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnz9s\" (UniqueName: \"kubernetes.io/projected/ccacf131-cba5-4c7d-8a6f-e8761c6aa67b-kube-api-access-xnz9s\") pod \"perf-node-gather-daemonset-22x9d\" (UID: \"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b\") " pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.785086 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.784777 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:54.968355 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:54.968317 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d"] Jul 10 18:44:54.971825 ip-10-0-141-217 kubenswrapper[2570]: W0710 18:44:54.971789 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podccacf131_cba5_4c7d_8a6f_e8761c6aa67b.slice/crio-90241e38c5956d18fd1c050ba3d92c6f8ac51a44ee008c691b64f68f2b248109 WatchSource:0}: Error finding container 90241e38c5956d18fd1c050ba3d92c6f8ac51a44ee008c691b64f68f2b248109: Status 404 returned error can't find the container with id 90241e38c5956d18fd1c050ba3d92c6f8ac51a44ee008c691b64f68f2b248109 Jul 10 18:44:55.043925 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:55.043842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" event={"ID":"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b","Type":"ContainerStarted","Data":"90241e38c5956d18fd1c050ba3d92c6f8ac51a44ee008c691b64f68f2b248109"} Jul 10 18:44:56.054688 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:56.054645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" event={"ID":"ccacf131-cba5-4c7d-8a6f-e8761c6aa67b","Type":"ContainerStarted","Data":"90e26020f52503d2e71144399e7e892fdbde91c93a1a4dec6fc314b34fd6a390"} Jul 10 18:44:56.055706 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:56.055678 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:44:56.072144 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:56.072078 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" podStartSLOduration=2.072057032 podStartE2EDuration="2.072057032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:44:56.071299729 +0000 UTC m=+2505.166490216" watchObservedRunningTime="2026-07-10 18:44:56.072057032 +0000 UTC m=+2505.167247518" Jul 10 18:44:56.554059 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:56.554028 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnzm2_0a0cf7f6-971f-4aef-be0d-8511986c673a/dns/0.log" Jul 10 18:44:56.590929 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:56.590898 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnzm2_0a0cf7f6-971f-4aef-be0d-8511986c673a/kube-rbac-proxy/0.log" Jul 10 18:44:56.717823 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:56.717791 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zt5vq_dbcb7ac8-3470-433d-9765-9cda8d1909f9/dns-node-resolver/0.log" Jul 10 18:44:57.287472 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:57.287434 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-685668c86c-fjk9j_34705db0-c7e8-4758-be6a-9867202042d1/registry/0.log" Jul 10 18:44:57.369097 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:57.369067 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hjstd_e8c2b25e-ffaf-48b9-b544-fff1ab93f1da/node-ca/0.log" Jul 10 18:44:58.530308 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:58.530278 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-655946c88b-vf5p4_dd0ff8f4-884b-4102-bd8a-409bad35e7b6/kube-auth-proxy/0.log" Jul 10 18:44:58.640928 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:58.640894 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-gmms5_6c08f902-e491-4458-ab6c-aade2b0e69d1/payload-pre-processing/0.log" Jul 10 18:44:59.424561 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:59.424528 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9drf9_381b690f-e076-4a9c-a3b9-a47422de7245/serve-healthcheck-canary/0.log" Jul 10 18:44:59.987042 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:44:59.987013 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22fz5_892cc365-9261-4418-bb36-bbdaa6bf1c4d/kube-rbac-proxy/0.log" Jul 10 18:45:00.012252 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:00.012220 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22fz5_892cc365-9261-4418-bb36-bbdaa6bf1c4d/exporter/0.log" Jul 10 18:45:00.042732 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:00.042699 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22fz5_892cc365-9261-4418-bb36-bbdaa6bf1c4d/extractor/0.log" Jul 10 18:45:02.581519 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:02.581421 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4jb82_1a43527f-276f-4d39-bf1d-3146106459a6/server/0.log" Jul 10 18:45:02.697338 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:02.697234 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ps68r_a3762dab-7ad1-440e-bbff-d8a6df59eccf/manager/0.log" Jul 10 18:45:02.834165 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:02.834051 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795b486df7-4bvw8_52a7d721-1131-4935-909b-043dd4f03e57/manager/0.log" Jul 10 18:45:02.854742 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:02.854714 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795b486df7-4bvw8_52a7d721-1131-4935-909b-043dd4f03e57/copy-manifests/0.log" Jul 10 18:45:03.085160 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:03.085081 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rjmm9/perf-node-gather-daemonset-22x9d" Jul 10 18:45:04.140181 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:04.140153 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59dbbcc5d9-qf4tn_44b5858e-5bee-459d-b8cb-56495557d930/manager/0.log" Jul 10 18:45:04.167930 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:04.167889 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pnm5z_c89a2afe-fdef-465e-91fe-bc0351080d9b/openshift-lws-operator/0.log" Jul 10 18:45:04.746375 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:04.746339 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-655946c88b-vf5p4_dd0ff8f4-884b-4102-bd8a-409bad35e7b6/kube-auth-proxy/0.log" Jul 10 18:45:04.835917 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:04.835875 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-gmms5_6c08f902-e491-4458-ab6c-aade2b0e69d1/payload-pre-processing/0.log" Jul 10 18:45:06.375029 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:06.374999 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-655946c88b-vf5p4_dd0ff8f4-884b-4102-bd8a-409bad35e7b6/kube-auth-proxy/0.log" Jul 10 18:45:06.456775 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:06.456732 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-gmms5_6c08f902-e491-4458-ab6c-aade2b0e69d1/payload-pre-processing/0.log" Jul 10 18:45:11.326101 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.326068 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwslj_32795b85-db8f-480d-9630-2d1d11a12bd3/kube-multus-additional-cni-plugins/0.log" Jul 10 18:45:11.349655 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.349621 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwslj_32795b85-db8f-480d-9630-2d1d11a12bd3/egress-router-binary-copy/0.log" Jul 10 18:45:11.376166 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.376138 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwslj_32795b85-db8f-480d-9630-2d1d11a12bd3/cni-plugins/0.log" Jul 10 18:45:11.399231 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.399201 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwslj_32795b85-db8f-480d-9630-2d1d11a12bd3/bond-cni-plugin/0.log" Jul 10 18:45:11.422157 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.422120 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwslj_32795b85-db8f-480d-9630-2d1d11a12bd3/routeoverride-cni/0.log" Jul 10 18:45:11.446419 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.446386 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwslj_32795b85-db8f-480d-9630-2d1d11a12bd3/whereabouts-cni-bincopy/0.log" Jul 10 18:45:11.472957 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.472925 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wwslj_32795b85-db8f-480d-9630-2d1d11a12bd3/whereabouts-cni/0.log" Jul 10 18:45:11.557034 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.557002 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sjwzz_8af2f4d1-f8be-47b7-b4e9-ed5d6d2be08f/kube-multus/0.log" Jul 10 18:45:11.716001 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.715967 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mgjzs_c04d6277-39ba-474e-af1e-3f61c91741c6/network-metrics-daemon/0.log" Jul 10 18:45:11.735180 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:11.735146 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mgjzs_c04d6277-39ba-474e-af1e-3f61c91741c6/kube-rbac-proxy/0.log" Jul 10 18:45:12.608052 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.608022 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-controller/0.log" Jul 10 18:45:12.627711 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.627672 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/1.log" Jul 10 18:45:12.627884 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.627720 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovn-acl-logging/0.log" Jul 10 18:45:12.647518 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.647482 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/kube-rbac-proxy-node/0.log" Jul 10 18:45:12.671027 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.671005 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 18:45:12.690180 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.690111 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/northd/0.log" Jul 10 18:45:12.713269 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.713242 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/nbdb/0.log" Jul 10 18:45:12.735538 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.735514 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/sbdb/0.log" Jul 10 18:45:12.819665 ip-10-0-141-217 kubenswrapper[2570]: I0710 18:45:12.819632 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6675h_974c1314-41cd-413c-adf4-0ca46ac529f4/ovnkube-controller/0.log"