Jul 07 17:20:31.073407 ip-10-0-129-95 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 17:20:31.073423 ip-10-0-129-95 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 17:20:31.073432 ip-10-0-129-95 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 17:20:31.073784 ip-10-0-129-95 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 17:20:41.095664 ip-10-0-129-95 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 17:20:41.095683 ip-10-0-129-95 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 86d70b5ee16844339ea9cf49cb3c1011 -- Jul 07 17:22:45.036598 ip-10-0-129-95 systemd[1]: Starting Kubernetes Kubelet... Jul 07 17:22:45.456524 ip-10-0-129-95 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:22:45.456524 ip-10-0-129-95 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 17:22:45.456524 ip-10-0-129-95 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:22:45.456524 ip-10-0-129-95 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 17:22:45.456524 ip-10-0-129-95 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:22:45.457668 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.457528 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 17:22:45.461928 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461912 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:45.461928 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461928 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461932 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461936 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461939 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461942 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461945 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461949 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461952 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461954 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461957 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461960 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461963 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461965 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461968 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461970 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461972 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461975 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461977 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461987 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461989 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:45.461991 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461992 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461994 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.461997 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462000 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462003 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462006 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462008 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462011 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462013 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462016 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462018 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462021 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462023 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462025 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462028 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462030 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462032 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462035 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462037 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:45.462511 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462041 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462045 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462048 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462051 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462053 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462056 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462059 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462062 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462064 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462067 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462069 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462072 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462074 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462078 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462081 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462084 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462086 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462089 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462091 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462094 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:45.462957 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462096 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462099 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462101 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462104 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462106 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462108 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462111 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462113 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462115 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462118 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462120 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462124 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462127 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462130 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462132 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462135 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462139 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462142 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462145 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:45.463447 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462149 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462152 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462155 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462158 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462161 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462165 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462168 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462171 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462173 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462175 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462177 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462180 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462183 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462185 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462187 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462190 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462192 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462195 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462197 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462200 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:45.463895 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462913 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462921 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462925 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462929 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462931 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462934 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462937 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462939 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462942 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462946 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462950 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462953 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462955 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462958 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462962 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462964 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462967 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462970 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462972 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:45.464387 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462975 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462977 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462980 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462982 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462985 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462987 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462990 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462992 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.462996 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463000 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463003 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463005 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463008 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463010 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463012 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463015 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463017 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463019 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463022 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:45.464850 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463024 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463027 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463029 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463032 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463035 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463037 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463039 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463042 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463044 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463047 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463049 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463052 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463054 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463057 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463059 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463062 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463065 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463067 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463069 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463072 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:45.465336 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463074 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463076 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463079 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463081 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463083 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463086 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463088 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463090 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463092 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463095 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463097 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463099 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463102 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463104 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463106 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463110 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463112 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463115 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463117 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463120 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:45.465812 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463122 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463124 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463127 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463129 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463132 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463134 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463139 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463141 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463143 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463146 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463148 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463150 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463153 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463155 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463158 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463160 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463162 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463164 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463167 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463169 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:45.466324 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463171 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463558 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463563 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463567 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463570 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463572 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463574 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463580 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463583 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463587 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463590 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463592 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463595 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463598 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463601 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463604 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463606 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463609 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463613 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:45.466808 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463615 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463618 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463620 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463623 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463625 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463627 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463630 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463632 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463634 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463637 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463640 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463643 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463645 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463648 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463650 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463653 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463655 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463658 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463660 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463662 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:45.467261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463665 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463667 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463670 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463672 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463675 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463677 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463680 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463682 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463685 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463687 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463690 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463695 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463698 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463701 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463704 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463707 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463709 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463712 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463715 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463717 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:45.467786 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463720 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463722 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463725 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463728 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463730 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463733 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463736 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463739 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463742 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463744 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463749 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463752 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463756 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463759 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463762 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463764 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463767 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463769 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463771 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463774 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:45.468261 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463776 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463779 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463781 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463785 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463787 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463790 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463792 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463794 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463797 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463799 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463801 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463804 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463806 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463808 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463811 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463813 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463815 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463817 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463820 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463822 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:45.468805 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.463824 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463889 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463895 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463901 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463908 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463912 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463915 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463919 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463923 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463926 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463928 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463932 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463935 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463937 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463941 2579 flags.go:64] FLAG: --cgroup-root="" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463945 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463948 2579 flags.go:64] FLAG: --client-ca-file="" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463951 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463953 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463958 2579 flags.go:64] FLAG: --cluster-domain="" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463960 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463963 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463966 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463969 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 17:22:45.469295 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463972 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463975 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463978 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463981 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463984 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463986 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463989 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463992 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463994 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.463998 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464001 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464004 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464007 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464010 2579 flags.go:64] FLAG: --enable-server="true" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464013 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464017 2579 flags.go:64] FLAG: --event-burst="100" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464020 2579 flags.go:64] FLAG: --event-qps="50" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464023 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464025 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464028 2579 flags.go:64] FLAG: --eviction-hard="" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464031 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464034 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464036 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464041 2579 flags.go:64] FLAG: --eviction-soft="" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464044 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 17:22:45.469869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464047 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464049 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464052 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464055 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464057 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464060 2579 flags.go:64] FLAG: --feature-gates="" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464063 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464066 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464069 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464072 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464075 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464077 2579 flags.go:64] FLAG: --help="false" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464080 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464083 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464086 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464088 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464091 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464095 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464097 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464101 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464104 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464106 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464109 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464112 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 17:22:45.470493 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464115 2579 flags.go:64] FLAG: --kube-reserved="" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464117 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464120 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464123 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464125 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464128 2579 flags.go:64] FLAG: --lock-file="" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464131 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464135 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464138 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464142 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464145 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464148 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464150 2579 flags.go:64] FLAG: --logging-format="text" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464153 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464156 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464158 2579 flags.go:64] FLAG: --manifest-url="" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464161 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464165 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464167 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464171 2579 flags.go:64] FLAG: --max-pods="110" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464174 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464176 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464179 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464181 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464184 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 17:22:45.471068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464187 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464190 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464197 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464200 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464203 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464206 2579 flags.go:64] FLAG: --pod-cidr="" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464208 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464213 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464216 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464218 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464221 2579 flags.go:64] FLAG: --port="10250" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464224 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464227 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0373b15d6f964c4ce" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464231 2579 flags.go:64] FLAG: --qos-reserved="" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464234 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464237 2579 flags.go:64] FLAG: --register-node="true" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464240 2579 flags.go:64] FLAG: --register-with-taints="" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464243 2579 flags.go:64] FLAG: --registry-burst="10" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464245 2579 flags.go:64] FLAG: --registry-qps="5" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464248 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464251 2579 flags.go:64] FLAG: --reserved-memory="" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464254 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464257 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464259 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464278 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 17:22:45.471683 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464281 2579 flags.go:64] FLAG: --runonce="false" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464284 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464287 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464289 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464292 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464295 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464298 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464301 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464303 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464306 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464310 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464312 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464315 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464318 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464321 2579 flags.go:64] FLAG: --system-cgroups="" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464323 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464328 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464331 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464333 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464337 2579 flags.go:64] FLAG: --tls-min-version="" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464341 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464344 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464347 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464350 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464353 2579 flags.go:64] FLAG: --v="2" Jul 07 17:22:45.472437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464356 2579 flags.go:64] FLAG: --version="false" Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464360 2579 flags.go:64] FLAG: --vmodule="" Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464363 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.464366 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464472 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464475 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464478 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464481 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464484 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464487 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464490 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464493 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464496 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464498 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464501 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464503 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464505 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464509 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464512 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464514 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464517 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:45.473031 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464520 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464522 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464525 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464527 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464530 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464532 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464535 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464538 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464541 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464543 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464546 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464548 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464551 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464553 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464555 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464558 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464561 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464563 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464566 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464568 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:45.473540 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464570 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464573 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464575 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464577 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464580 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464582 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464584 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464587 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464591 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464594 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464596 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464599 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464601 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464604 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464606 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464609 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464611 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464613 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464617 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464620 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:45.474117 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464623 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464627 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464630 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464633 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464636 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464638 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464640 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464643 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464645 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464648 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464650 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464652 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464655 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464657 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464661 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464664 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464666 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464669 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:45.474618 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464671 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464675 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464680 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464699 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464703 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464706 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464709 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464712 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464715 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464718 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464720 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464723 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464727 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464730 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464732 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464735 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464737 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464740 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464743 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:45.475042 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464745 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464748 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464750 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464752 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.464755 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.465778 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.473392 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.473405 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.473423 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.473432 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473483 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473488 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473491 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473494 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473497 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:45.475516 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473500 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473502 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473505 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473507 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473510 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473512 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473515 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473517 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473520 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473522 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473525 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473528 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473530 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473533 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473535 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473538 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473540 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473543 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473546 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473548 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:45.475881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473551 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473553 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473556 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473559 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473562 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473564 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473567 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473569 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473572 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473574 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473577 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473579 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473582 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473586 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473590 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473593 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473596 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473598 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473601 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473604 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:45.476382 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473606 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473608 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473611 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473614 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473616 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473618 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473621 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473623 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473625 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473628 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473631 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473633 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473636 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473638 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473641 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473644 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473647 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473649 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473652 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473655 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:45.476897 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473657 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473660 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473662 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473664 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473667 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473670 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473672 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473676 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473679 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473683 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473686 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473688 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473691 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473693 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473695 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473698 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473701 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473705 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473708 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:45.477392 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473711 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473714 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473717 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473719 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473722 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473724 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473727 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473730 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473732 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473735 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473738 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473740 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473743 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473745 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473747 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:45.477965 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.473752 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473857 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473861 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473863 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473866 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473869 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473872 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473874 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473876 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473879 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473881 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473883 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473885 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473888 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473890 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473892 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473895 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473897 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473900 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473902 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:22:45.478348 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473904 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473907 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473910 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473912 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473914 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473917 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473921 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473924 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473927 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473929 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473931 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473934 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473936 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473939 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473941 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473943 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473946 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473948 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473950 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473953 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:22:45.478861 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473955 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473959 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473962 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473965 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473968 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473971 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473973 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473976 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473978 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473981 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473984 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473986 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473989 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473992 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.473998 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474000 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474004 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474007 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474010 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:22:45.479442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474013 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474015 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474018 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474020 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474023 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474025 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474028 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474030 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474033 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474036 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474038 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474041 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474044 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474046 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474049 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474052 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474054 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474057 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474059 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474062 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:22:45.480019 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474064 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474067 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474069 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474071 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474073 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474076 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474078 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474081 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474084 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474086 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474089 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474091 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474094 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474097 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474099 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474102 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474104 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474107 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474109 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:22:45.480834 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474111 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:45.474114 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.474118 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.474203 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.476200 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.477620 2579 server.go:1019] "Starting client certificate rotation" Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.477742 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.477794 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:22:45.481365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.480498 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:22:45.483372 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.483351 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:22:45.498719 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.498700 2579 log.go:25] "Validated CRI v1 runtime API" Jul 07 17:22:45.500665 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.500652 2579 log.go:25] "Validated CRI v1 image API" Jul 07 17:22:45.501477 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.501466 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 17:22:45.504631 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.504615 2579 fs.go:135] Filesystem UUIDs: map[4982c77a-ea6f-4529-a1fc-150e79ee8a5a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 933c6792-0a00-43aa-bcc9-916cdf23ea60:/dev/nvme0n1p4] Jul 07 17:22:45.504684 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.504632 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 17:22:45.510458 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.510347 2579 manager.go:216] Machine: {Timestamp:2026-07-07 17:22:45.508934734 +0000 UTC m=+0.349248397 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100151 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2102b7d958f3993fd3116154dba3ce SystemUUID:ec2102b7-d958-f399-3fd3-116154dba3ce BootID:86d70b5e-e168-4433-9ea9-cf49cb3c1011 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b1:db:dd:6b:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b1:db:dd:6b:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:3b:be:7b:dc:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 17:22:45.510458 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.510451 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 17:22:45.510569 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.510525 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 17:22:45.512058 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.512036 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:22:45.513972 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.513947 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 17:22:45.514100 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.513976 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-95.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 17:22:45.515525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.515515 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 17:22:45.515558 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.515528 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 17:22:45.515558 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.515536 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:22:45.516142 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.516132 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:22:45.517446 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.517435 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 17:22:45.520132 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520121 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:22:45.520236 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520227 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 17:22:45.520294 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520256 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 07 17:22:45.520294 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520289 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 17:22:45.520365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520302 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 17:22:45.520365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520313 2579 kubelet.go:398] "Adding apiserver pod source" Jul 07 17:22:45.520365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520321 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 17:22:45.520648 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520637 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:22:45.520682 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.520655 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:22:45.522792 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.522777 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 17:22:45.524523 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.524510 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 17:22:45.524600 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.524539 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 17:22:45.525562 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525545 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 17:22:45.525562 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525562 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525594 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525601 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525613 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525620 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525629 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525636 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525644 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 17:22:45.525658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525650 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 17:22:45.525860 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525674 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 17:22:45.525860 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.525683 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 17:22:45.526865 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.526855 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 17:22:45.526865 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.526864 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 17:22:45.529973 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.529960 2579 server.go:1268] "Started kubelet" Jul 07 17:22:45.530092 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.530037 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 17:22:45.530148 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.530051 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 17:22:45.530148 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.530117 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 17:22:45.530739 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.530683 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-95.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 17:22:45.530739 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.530687 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 17:22:45.530878 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.530809 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 17:22:45.530952 ip-10-0-129-95 systemd[1]: Started Kubernetes Kubelet. Jul 07 17:22:45.531747 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.531712 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 17:22:45.536373 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.536347 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 07 17:22:45.537749 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.537708 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 17:22:45.540650 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.540632 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxpwz" Jul 07 17:22:45.542324 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.541213 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-95.ec2.internal.18c011bf3ededa02 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-95.ec2.internal,UID:ip-10-0-129-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-95.ec2.internal,},FirstTimestamp:2026-07-07 17:22:45.529934338 +0000 UTC m=+0.370247999,LastTimestamp:2026-07-07 17:22:45.529934338 +0000 UTC m=+0.370247999,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-95.ec2.internal,}" Jul 07 17:22:45.542847 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.542833 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 17:22:45.542909 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.542854 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 17:22:45.542975 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.542956 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:45.543413 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543397 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 17:22:45.543550 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543474 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 17:22:45.543611 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543561 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 17:22:45.543685 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543669 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 07 17:22:45.543685 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543682 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 07 17:22:45.543937 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543923 2579 factory.go:153] Registering CRI-O factory Jul 07 17:22:45.544017 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543942 2579 factory.go:223] Registration of the crio container factory successfully Jul 07 17:22:45.544017 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.543992 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 17:22:45.544017 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.544000 2579 factory.go:55] Registering systemd factory Jul 07 17:22:45.544017 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.544008 2579 factory.go:223] Registration of the systemd container factory successfully Jul 07 17:22:45.544203 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.544029 2579 factory.go:103] Registering Raw factory Jul 07 17:22:45.544203 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.544043 2579 manager.go:1196] Started watching for new ooms in manager Jul 07 17:22:45.544474 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.544459 2579 manager.go:349] Starting recovery of all containers Jul 07 17:22:45.547659 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.547631 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 17:22:45.547815 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.547794 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 17:22:45.551863 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.551725 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxpwz" Jul 07 17:22:45.552059 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.552046 2579 manager.go:354] Recovery completed Jul 07 17:22:45.556119 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.556103 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:45.557781 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.557735 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:45.557781 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.557769 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:45.557781 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.557784 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:45.558242 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.558230 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 17:22:45.558242 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.558240 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 17:22:45.558341 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.558254 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:22:45.558385 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.558372 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:45.559032 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.559019 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:45.559088 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.559043 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:45.559088 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.559053 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:45.559370 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.559359 2579 policy_none.go:49] "None policy: Start" Jul 07 17:22:45.559411 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.559375 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 17:22:45.559411 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.559385 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 17:22:45.560373 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.560205 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-95.ec2.internal.18c011bf40875a0b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-95.ec2.internal,UID:ip-10-0-129-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-95.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-95.ec2.internal,},FirstTimestamp:2026-07-07 17:22:45.557754379 +0000 UTC m=+0.398068046,LastTimestamp:2026-07-07 17:22:45.557754379 +0000 UTC m=+0.398068046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-95.ec2.internal,}" Jul 07 17:22:45.560373 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.560315 2579 policy_none.go:47] "Start" Jul 07 17:22:45.611836 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.611822 2579 manager.go:337] "Starting Device Plugin manager" Jul 07 17:22:45.611908 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.611853 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 17:22:45.611908 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.611863 2579 server.go:87] "Starting device plugin registration server" Jul 07 17:22:45.612126 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.612113 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 17:22:45.612126 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.612126 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 17:22:45.612251 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.612238 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 17:22:45.612353 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.612336 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 17:22:45.612353 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.612353 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 17:22:45.612778 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.612760 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 17:22:45.612847 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.612811 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:45.712481 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.712441 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:45.713553 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.713361 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:45.713553 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.713389 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:45.713553 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.713404 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:45.713553 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.713431 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.721739 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.721727 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.721787 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.721745 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-95.ec2.internal\": node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:45.735387 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.735369 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:45.834686 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.834655 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 17:22:45.835966 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.835941 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:45.836069 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.836031 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 17:22:45.836106 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.836079 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 17:22:45.836106 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.836105 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 17:22:45.836187 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.836147 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 17:22:45.838670 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.838646 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:45.936980 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.936935 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:45.936980 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.936956 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"] Jul 07 17:22:45.937183 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.937043 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:45.940061 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.940042 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:45.940169 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.940077 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:45.940169 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.940091 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:45.941424 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.941409 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:45.941530 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.941516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.941582 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.941548 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:45.942165 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.942148 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:45.942234 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.942176 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:45.942234 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.942185 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:45.942355 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.942153 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:45.942355 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.942257 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:45.942355 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.942294 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:45.943404 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.943389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.943483 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.943416 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:22:45.944558 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.944536 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:22:45.944646 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.944579 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:22:45.944646 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.944595 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:22:45.944828 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.944812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eff5104d7452b5ed4a3bf2742a0cf695-config\") pod \"kube-apiserver-proxy-ip-10-0-129-95.ec2.internal\" (UID: \"eff5104d7452b5ed4a3bf2742a0cf695\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.944863 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.944843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09305b88f1373ed8937b4f40e869c133-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"09305b88f1373ed8937b4f40e869c133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.944899 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:45.944874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09305b88f1373ed8937b4f40e869c133-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"09305b88f1373ed8937b4f40e869c133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.968943 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.968890 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-95.ec2.internal\" not found" node="ip-10-0-129-95.ec2.internal" Jul 07 17:22:45.973539 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:45.973521 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-95.ec2.internal\" not found" node="ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.037520 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.037499 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.045834 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.045812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eff5104d7452b5ed4a3bf2742a0cf695-config\") pod \"kube-apiserver-proxy-ip-10-0-129-95.ec2.internal\" (UID: \"eff5104d7452b5ed4a3bf2742a0cf695\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.045922 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.045842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09305b88f1373ed8937b4f40e869c133-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"09305b88f1373ed8937b4f40e869c133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.045922 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.045867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09305b88f1373ed8937b4f40e869c133-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"09305b88f1373ed8937b4f40e869c133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.045995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.045917 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eff5104d7452b5ed4a3bf2742a0cf695-config\") pod \"kube-apiserver-proxy-ip-10-0-129-95.ec2.internal\" (UID: \"eff5104d7452b5ed4a3bf2742a0cf695\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.045995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.045919 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09305b88f1373ed8937b4f40e869c133-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"09305b88f1373ed8937b4f40e869c133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.045995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.045920 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09305b88f1373ed8937b4f40e869c133-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"09305b88f1373ed8937b4f40e869c133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.139972 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.139933 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.247756 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.247729 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.273471 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.273439 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.276719 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.276687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.348057 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.348006 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.448790 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.448740 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.477068 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.477037 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 17:22:46.477736 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.477210 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 17:22:46.549501 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.549418 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.554870 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.554834 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 17:17:46 +0000 UTC" deadline="2028-01-01 17:17:18.687563464 +0000 UTC" Jul 07 17:22:46.554916 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.554873 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13031h54m32.132695422s" Jul 07 17:22:46.650162 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.650127 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.696002 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.695973 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:46.751241 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:46.751204 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found" Jul 07 17:22:46.764672 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.764648 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:46.843839 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.843757 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.855356 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.855321 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:22:46.857224 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.857209 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" Jul 07 17:22:46.860460 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:46.860433 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09305b88f1373ed8937b4f40e869c133.slice/crio-6f93ba3ac6fb64cca2c0768457537e87f23d0e9bccc3fa3b833a966133d470e8 WatchSource:0}: Error finding container 6f93ba3ac6fb64cca2c0768457537e87f23d0e9bccc3fa3b833a966133d470e8: Status 404 returned error can't find the container with id 6f93ba3ac6fb64cca2c0768457537e87f23d0e9bccc3fa3b833a966133d470e8 Jul 07 17:22:46.865257 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.865240 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:22:46.873385 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:46.873365 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:22:46.892279 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:46.892245 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff5104d7452b5ed4a3bf2742a0cf695.slice/crio-780dc0aae56b94712f61dd57c06639c0add007c13a772637ec4b3e87d74d3c79 WatchSource:0}: Error finding container 780dc0aae56b94712f61dd57c06639c0add007c13a772637ec4b3e87d74d3c79: Status 404 returned error can't find the container with id 780dc0aae56b94712f61dd57c06639c0add007c13a772637ec4b3e87d74d3c79 Jul 07 17:22:47.052145 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.052119 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:47.521458 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.521424 2579 apiserver.go:52] "Watching apiserver" Jul 07 17:22:47.534321 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.534294 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 17:22:47.535323 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.535150 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-fhpv8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal","openshift-multus/multus-additional-cni-plugins-d7p99","openshift-network-diagnostics/network-check-target-f48zt","kube-system/global-pull-secret-syncer-29cf4","kube-system/konnectivity-agent-ttk6q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2","openshift-dns/node-resolver-mw4xf","openshift-multus/multus-ct9rk","openshift-multus/network-metrics-daemon-xkrbk","openshift-network-operator/iptables-alerter-xqxtf","openshift-ovn-kubernetes/ovnkube-node-4bsf6","kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hrhfd"] Jul 07 17:22:47.538052 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.538028 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.540542 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.540520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.541131 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.541107 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 17:22:47.541354 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.541332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 17:22:47.541727 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.541522 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tvd89\"" Jul 07 17:22:47.541727 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.541570 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 17:22:47.544946 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.543162 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 17:22:47.544946 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.543455 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:47.544946 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.543687 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:47.544946 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.543692 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b2zj8\"" Jul 07 17:22:47.544946 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.543871 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 17:22:47.544946 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.544394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.547828 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.547804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 17:22:47.547917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.547889 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 17:22:47.547917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.547899 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 17:22:47.548079 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.548064 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 17:22:47.548156 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.548138 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 17:22:47.548239 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.548221 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pkmrd\"" Jul 07 17:22:47.548374 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.548352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:47.548453 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.548423 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:22:47.550876 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.550702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.550876 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.550768 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:22:47.554044 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.554020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.555958 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.555934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06d0b994-7470-4e74-841d-4468df9db241-host-slash\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.555958 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.555954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.556109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.555969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-os-release\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.556109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.555997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.556109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-registration-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.556109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.556109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-sys-fs\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.556109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-system-cni-dir\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.556109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cnibin\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxsx\" (UniqueName: \"kubernetes.io/projected/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-kube-api-access-mbxsx\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-socket-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpgw\" (UniqueName: \"kubernetes.io/projected/06d0b994-7470-4e74-841d-4468df9db241-kube-api-access-fqpgw\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-device-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ptp\" (UniqueName: \"kubernetes.io/projected/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-kube-api-access-n9ptp\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06d0b994-7470-4e74-841d-4468df9db241-iptables-alerter-script\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.556491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.557034 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556739 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 17:22:47.557034 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.556758 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 17:22:47.557034 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.557000 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5n7xp\"" Jul 07 17:22:47.559242 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.559221 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 17:22:47.559512 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.559495 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzfxc\"" Jul 07 17:22:47.559764 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.559744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.562867 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.562138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.563063 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.563022 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:22:47.563288 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.563269 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8mkqb\"" Jul 07 17:22:47.563374 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.563318 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:22:47.563998 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.563979 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 17:22:47.564626 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.564598 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 17:22:47.564926 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.564903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:47.565018 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.564990 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:22:47.565283 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.565245 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 17:22:47.565393 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.565364 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gww52\"" Jul 07 17:22:47.567708 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.567241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.569985 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.569961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.571155 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.571139 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 17:22:47.571304 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.571292 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 17:22:47.571530 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.571513 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8bl4t\"" Jul 07 17:22:47.571779 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.571758 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 17:22:47.574629 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.574607 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 17:22:47.574993 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.574973 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 17:22:47.575268 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.575249 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 17:22:47.575752 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.575734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 17:22:47.575991 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.575975 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 17:22:47.576821 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.576801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 17:22:47.579783 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.577374 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hx5rw\"" Jul 07 17:22:47.644885 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.644862 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 17:22:47.646199 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.646176 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pbdqt" Jul 07 17:22:47.657070 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-kubernetes\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.657171 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-systemd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657171 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-etc-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657171 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-log-socket\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657171 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-env-overrides\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:47.657388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysctl-conf\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.657388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-var-lib-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-ovnkube-config\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/35236574-483a-4f91-8738-d99b59016097-ovn-node-metrics-cert\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-socket-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpgw\" (UniqueName: \"kubernetes.io/projected/06d0b994-7470-4e74-841d-4468df9db241-kube-api-access-fqpgw\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysctl-d\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-cni-netd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljnd\" (UniqueName: \"kubernetes.io/projected/35236574-483a-4f91-8738-d99b59016097-kube-api-access-zljnd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-socket-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ptp\" (UniqueName: \"kubernetes.io/projected/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-kube-api-access-n9ptp\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06d0b994-7470-4e74-841d-4468df9db241-iptables-alerter-script\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-cnibin\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysconfig\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-slash\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06d0b994-7470-4e74-841d-4468df9db241-host-slash\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.657738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/509edf3c-5fd7-4e16-85b4-7405c98a2900-konnectivity-ca\") pod \"konnectivity-agent-ttk6q\" (UID: \"509edf3c-5fd7-4e16-85b4-7405c98a2900\") " pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvs9\" (UniqueName: \"kubernetes.io/projected/955bef82-3b6a-47dd-9a19-113ab9aaebee-kube-api-access-spvs9\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-sys\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74a13726-6b26-4315-baa7-68a633a5a363-etc-tuned\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-sys-fs\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06d0b994-7470-4e74-841d-4468df9db241-host-slash\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxsx\" (UniqueName: \"kubernetes.io/projected/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-kube-api-access-mbxsx\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-modprobe-d\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.657974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-lib-modules\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658097 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658104 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06d0b994-7470-4e74-841d-4468df9db241-iptables-alerter-script\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658169 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-sys-fs\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-kubelet\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.658361 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/509edf3c-5fd7-4e16-85b4-7405c98a2900-agent-certs\") pod \"konnectivity-agent-ttk6q\" (UID: \"509edf3c-5fd7-4e16-85b4-7405c98a2900\") " pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-var-lib-kubelet\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-ovnkube-script-lib\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8r9\" (UniqueName: \"kubernetes.io/projected/368511c4-3427-4707-9c1f-7e5d31d53fb4-kube-api-access-cs8r9\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-os-release\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-netns\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-cni-bin\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-hostroot\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnmd\" (UniqueName: \"kubernetes.io/projected/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-kube-api-access-ptnmd\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-cni-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66c0087b-784a-460f-a00d-b2384a9eae45-cni-binary-copy\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-cni-multus\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-systemd\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sm8n\" (UniqueName: \"kubernetes.io/projected/74a13726-6b26-4315-baa7-68a633a5a363-kube-api-access-5sm8n\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658722 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.659181 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-run-ovn-kubernetes\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cnibin\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658846 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cnibin\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-etc-kubernetes\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.658986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/955bef82-3b6a-47dd-9a19-113ab9aaebee-hosts-file\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-socket-dir-parent\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-run\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-device-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955bef82-3b6a-47dd-9a19-113ab9aaebee-tmp-dir\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/368511c4-3427-4707-9c1f-7e5d31d53fb4-serviceca\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-system-cni-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-device-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-kubelet\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-node-log\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.660095 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659245 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-os-release\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwdh\" (UniqueName: \"kubernetes.io/projected/66c0087b-784a-460f-a00d-b2384a9eae45-kube-api-access-sbwdh\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659329 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-os-release\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-host\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-ovn\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-registration-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-system-cni-dir\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de81cac2-c315-4825-927b-2b4c4f8f3790-dbus\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-k8s-cni-cncf-io\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74a13726-6b26-4315-baa7-68a633a5a363-tmp\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659544 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-system-cni-dir\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de81cac2-c315-4825-927b-2b4c4f8f3790-kubelet-config\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-registration-dir\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368511c4-3427-4707-9c1f-7e5d31d53fb4-host\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-multus-certs\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-systemd-units\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.660920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-run-netns\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-cni-bin\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-conf-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66c0087b-784a-460f-a00d-b2384a9eae45-multus-daemon-config\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.659975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.660618 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.661761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.660929 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pbdqt" Jul 07 17:22:47.665837 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.665816 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:47.665924 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.665847 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:47.665924 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.665873 2579 projected.go:196] Error preparing data for projected volume kube-api-access-skt8n for pod openshift-network-diagnostics/network-check-target-f48zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:47.666026 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.665948 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n podName:5298398f-faa0-40fb-b6c8-420e6a8c7fb3 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:48.165915907 +0000 UTC m=+3.006229558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-skt8n" (UniqueName: "kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n") pod "network-check-target-f48zt" (UID: "5298398f-faa0-40fb-b6c8-420e6a8c7fb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:47.667605 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.667343 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 17:22:47.671883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.671817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpgw\" (UniqueName: \"kubernetes.io/projected/06d0b994-7470-4e74-841d-4468df9db241-kube-api-access-fqpgw\") pod \"iptables-alerter-xqxtf\" (UID: \"06d0b994-7470-4e74-841d-4468df9db241\") " pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.671883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.671841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxsx\" (UniqueName: \"kubernetes.io/projected/e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8-kube-api-access-mbxsx\") pod \"multus-additional-cni-plugins-d7p99\" (UID: \"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8\") " pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.671883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.671848 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ptp\" (UniqueName: \"kubernetes.io/projected/18fd592a-667c-4d50-8d14-c1fcacf2c2bc-kube-api-access-n9ptp\") pod \"aws-ebs-csi-driver-node-8qdp2\" (UID: \"18fd592a-667c-4d50-8d14-c1fcacf2c2bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.760175 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de81cac2-c315-4825-927b-2b4c4f8f3790-kubelet-config\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.760327 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368511c4-3427-4707-9c1f-7e5d31d53fb4-host\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.760327 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-multus-certs\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.760327 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-systemd-units\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760327 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-run-netns\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760327 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-cni-bin\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-conf-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66c0087b-784a-460f-a00d-b2384a9eae45-multus-daemon-config\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-kubernetes\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-systemd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-etc-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-log-socket\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-env-overrides\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysctl-conf\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.760551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-var-lib-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-ovnkube-config\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/35236574-483a-4f91-8738-d99b59016097-ovn-node-metrics-cert\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysctl-d\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-cni-netd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljnd\" (UniqueName: \"kubernetes.io/projected/35236574-483a-4f91-8738-d99b59016097-kube-api-access-zljnd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-cnibin\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysconfig\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-slash\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/509edf3c-5fd7-4e16-85b4-7405c98a2900-konnectivity-ca\") pod \"konnectivity-agent-ttk6q\" (UID: \"509edf3c-5fd7-4e16-85b4-7405c98a2900\") " pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvs9\" (UniqueName: \"kubernetes.io/projected/955bef82-3b6a-47dd-9a19-113ab9aaebee-kube-api-access-spvs9\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-sys\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.760879 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74a13726-6b26-4315-baa7-68a633a5a363-etc-tuned\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-modprobe-d\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-lib-modules\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.760985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-kubelet\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/509edf3c-5fd7-4e16-85b4-7405c98a2900-agent-certs\") pod \"konnectivity-agent-ttk6q\" (UID: \"509edf3c-5fd7-4e16-85b4-7405c98a2900\") " pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-var-lib-kubelet\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-ovnkube-script-lib\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8r9\" (UniqueName: \"kubernetes.io/projected/368511c4-3427-4707-9c1f-7e5d31d53fb4-kube-api-access-cs8r9\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-os-release\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-netns\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761190 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66c0087b-784a-460f-a00d-b2384a9eae45-multus-daemon-config\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761256 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-systemd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761297 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761268 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-kubernetes\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761308 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-etc-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de81cac2-c315-4825-927b-2b4c4f8f3790-kubelet-config\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761374 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-log-socket\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761382 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368511c4-3427-4707-9c1f-7e5d31d53fb4-host\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761423 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-systemd-units\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761463 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-run-netns\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761502 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-cni-bin\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761539 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-conf-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-cni-bin\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-hostroot\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnmd\" (UniqueName: \"kubernetes.io/projected/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-kube-api-access-ptnmd\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-cni-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66c0087b-784a-460f-a00d-b2384a9eae45-cni-binary-copy\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-cni-multus\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-systemd\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sm8n\" (UniqueName: \"kubernetes.io/projected/74a13726-6b26-4315-baa7-68a633a5a363-kube-api-access-5sm8n\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-run-ovn-kubernetes\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.761883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-etc-kubernetes\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761845 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-env-overrides\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/955bef82-3b6a-47dd-9a19-113ab9aaebee-hosts-file\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-socket-dir-parent\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-run\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955bef82-3b6a-47dd-9a19-113ab9aaebee-tmp-dir\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761904 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-multus-certs\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761942 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-cni-bin\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.761964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-hostroot\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysctl-conf\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/368511c4-3427-4707-9c1f-7e5d31d53fb4-serviceca\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-system-cni-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-kubelet\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-node-log\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwdh\" (UniqueName: \"kubernetes.io/projected/66c0087b-784a-460f-a00d-b2384a9eae45-kube-api-access-sbwdh\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-cni-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.762612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-host\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-ovn\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de81cac2-c315-4825-927b-2b4c4f8f3790-dbus\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-k8s-cni-cncf-io\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74a13726-6b26-4315-baa7-68a633a5a363-tmp\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762561 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66c0087b-784a-460f-a00d-b2384a9eae45-cni-binary-copy\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762615 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-cni-multus\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762666 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-systemd\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762801 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-run-ovn-kubernetes\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-etc-kubernetes\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/955bef82-3b6a-47dd-9a19-113ab9aaebee-hosts-file\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-multus-socket-dir-parent\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.762960 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.762978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-run\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.763030 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs podName:0cba86e4-b5d0-446d-938b-ccf8dbc16c59 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:48.263010521 +0000 UTC m=+3.103324179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs") pod "network-metrics-daemon-xkrbk" (UID: "0cba86e4-b5d0-446d-938b-ccf8dbc16c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:47.763654 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.763210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-var-lib-openvswitch\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.764506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.763237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955bef82-3b6a-47dd-9a19-113ab9aaebee-tmp-dir\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.764506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.763619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/368511c4-3427-4707-9c1f-7e5d31d53fb4-serviceca\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.764506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.763673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-kubelet\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.764506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.763693 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-ovnkube-config\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.764506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.764233 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-system-cni-dir\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.764506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.764284 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-var-lib-kubelet\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.764506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.764327 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-node-log\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.764838 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.764561 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/509edf3c-5fd7-4e16-85b4-7405c98a2900-konnectivity-ca\") pod \"konnectivity-agent-ttk6q\" (UID: \"509edf3c-5fd7-4e16-85b4-7405c98a2900\") " pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.764838 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.764580 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-var-lib-kubelet\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.764838 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.764698 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysctl-d\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.764838 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.764749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-cni-netd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.765097 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765077 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/35236574-483a-4f91-8738-d99b59016097-ovnkube-script-lib\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.765163 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-cnibin\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.765163 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765142 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-run-ovn\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.765270 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-sysconfig\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.765270 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765192 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-host\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.765270 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765221 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35236574-483a-4f91-8738-d99b59016097-host-slash\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.765270 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-os-release\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.765270 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765266 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-netns\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.765547 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765316 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-sys\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.765547 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.765332 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:47.765547 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:47.765454 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret podName:de81cac2-c315-4825-927b-2b4c4f8f3790 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:48.265437841 +0000 UTC m=+3.105751496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret") pod "global-pull-secret-syncer-29cf4" (UID: "de81cac2-c315-4825-927b-2b4c4f8f3790") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:47.765697 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765629 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-lib-modules\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.765752 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74a13726-6b26-4315-baa7-68a633a5a363-etc-modprobe-d\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.765845 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de81cac2-c315-4825-927b-2b4c4f8f3790-dbus\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:47.765905 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.765879 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66c0087b-784a-460f-a00d-b2384a9eae45-host-run-k8s-cni-cncf-io\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.768155 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.768131 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/35236574-483a-4f91-8738-d99b59016097-ovn-node-metrics-cert\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.768477 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.768457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74a13726-6b26-4315-baa7-68a633a5a363-tmp\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.769491 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.769303 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:22:47.769598 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.769532 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/509edf3c-5fd7-4e16-85b4-7405c98a2900-agent-certs\") pod \"konnectivity-agent-ttk6q\" (UID: \"509edf3c-5fd7-4e16-85b4-7405c98a2900\") " pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.770741 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.770717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74a13726-6b26-4315-baa7-68a633a5a363-etc-tuned\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.775426 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.775362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sm8n\" (UniqueName: \"kubernetes.io/projected/74a13726-6b26-4315-baa7-68a633a5a363-kube-api-access-5sm8n\") pod \"tuned-hrhfd\" (UID: \"74a13726-6b26-4315-baa7-68a633a5a363\") " pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.775725 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.775539 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnmd\" (UniqueName: \"kubernetes.io/projected/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-kube-api-access-ptnmd\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:47.776825 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.776797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvs9\" (UniqueName: \"kubernetes.io/projected/955bef82-3b6a-47dd-9a19-113ab9aaebee-kube-api-access-spvs9\") pod \"node-resolver-mw4xf\" (UID: \"955bef82-3b6a-47dd-9a19-113ab9aaebee\") " pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.778052 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.777702 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwdh\" (UniqueName: \"kubernetes.io/projected/66c0087b-784a-460f-a00d-b2384a9eae45-kube-api-access-sbwdh\") pod \"multus-ct9rk\" (UID: \"66c0087b-784a-460f-a00d-b2384a9eae45\") " pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.778052 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.778022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljnd\" (UniqueName: \"kubernetes.io/projected/35236574-483a-4f91-8738-d99b59016097-kube-api-access-zljnd\") pod \"ovnkube-node-4bsf6\" (UID: \"35236574-483a-4f91-8738-d99b59016097\") " pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.779792 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.779750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8r9\" (UniqueName: \"kubernetes.io/projected/368511c4-3427-4707-9c1f-7e5d31d53fb4-kube-api-access-cs8r9\") pod \"node-ca-fhpv8\" (UID: \"368511c4-3427-4707-9c1f-7e5d31d53fb4\") " pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.841943 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.841888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" event={"ID":"eff5104d7452b5ed4a3bf2742a0cf695","Type":"ContainerStarted","Data":"780dc0aae56b94712f61dd57c06639c0add007c13a772637ec4b3e87d74d3c79"} Jul 07 17:22:47.843193 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.843133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" event={"ID":"09305b88f1373ed8937b4f40e869c133","Type":"ContainerStarted","Data":"6f93ba3ac6fb64cca2c0768457537e87f23d0e9bccc3fa3b833a966133d470e8"} Jul 07 17:22:47.852775 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.852754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" Jul 07 17:22:47.864389 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.864197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqxtf" Jul 07 17:22:47.865235 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:47.865209 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fd592a_667c_4d50_8d14_c1fcacf2c2bc.slice/crio-7b0aaa12085a5eded978c7ab93d1a474cffc90e3dd82bf8e9dcc51608d1a6991 WatchSource:0}: Error finding container 7b0aaa12085a5eded978c7ab93d1a474cffc90e3dd82bf8e9dcc51608d1a6991: Status 404 returned error can't find the container with id 7b0aaa12085a5eded978c7ab93d1a474cffc90e3dd82bf8e9dcc51608d1a6991 Jul 07 17:22:47.877525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.877472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7p99" Jul 07 17:22:47.887175 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.887153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:22:47.896182 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.896159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ct9rk" Jul 07 17:22:47.901915 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:47.901892 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509edf3c_5fd7_4e16_85b4_7405c98a2900.slice/crio-4d7f7bbf8749ecf92989e0b9fff1720c7312fc0331d53f514ba8f08c466d63f2 WatchSource:0}: Error finding container 4d7f7bbf8749ecf92989e0b9fff1720c7312fc0331d53f514ba8f08c466d63f2: Status 404 returned error can't find the container with id 4d7f7bbf8749ecf92989e0b9fff1720c7312fc0331d53f514ba8f08c466d63f2 Jul 07 17:22:47.904162 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.903877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" Jul 07 17:22:47.907903 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:47.907866 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c0087b_784a_460f_a00d_b2384a9eae45.slice/crio-aecc5e31a7db61638bc0601c795df3eb3880123a4dc22e9e1dfc7f348802dfe1 WatchSource:0}: Error finding container aecc5e31a7db61638bc0601c795df3eb3880123a4dc22e9e1dfc7f348802dfe1: Status 404 returned error can't find the container with id aecc5e31a7db61638bc0601c795df3eb3880123a4dc22e9e1dfc7f348802dfe1 Jul 07 17:22:47.914833 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.914812 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mw4xf" Jul 07 17:22:47.915386 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:47.915355 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a13726_6b26_4315_baa7_68a633a5a363.slice/crio-5781644c5807d63479f1535ce9b45c9e40f73fc967e3c7efe5c77abf4464a1eb WatchSource:0}: Error finding container 5781644c5807d63479f1535ce9b45c9e40f73fc967e3c7efe5c77abf4464a1eb: Status 404 returned error can't find the container with id 5781644c5807d63479f1535ce9b45c9e40f73fc967e3c7efe5c77abf4464a1eb Jul 07 17:22:47.923232 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.923210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fhpv8" Jul 07 17:22:47.927433 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:47.927411 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955bef82_3b6a_47dd_9a19_113ab9aaebee.slice/crio-c4439568cb9b20bb5ca2612a6912314d5acc95735d07901824bc21aa496d1633 WatchSource:0}: Error finding container c4439568cb9b20bb5ca2612a6912314d5acc95735d07901824bc21aa496d1633: Status 404 returned error can't find the container with id c4439568cb9b20bb5ca2612a6912314d5acc95735d07901824bc21aa496d1633 Jul 07 17:22:47.929754 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:47.929692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:22:47.934134 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:47.934105 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368511c4_3427_4707_9c1f_7e5d31d53fb4.slice/crio-22927f0a72b07809d2f820578bfc7eb3b58f815dbb921367443f104710c19037 WatchSource:0}: Error finding container 22927f0a72b07809d2f820578bfc7eb3b58f815dbb921367443f104710c19037: Status 404 returned error can't find the container with id 22927f0a72b07809d2f820578bfc7eb3b58f815dbb921367443f104710c19037 Jul 07 17:22:47.941611 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:22:47.941429 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35236574_483a_4f91_8738_d99b59016097.slice/crio-72fcae9f2c71a4ef3f5acafe3ae70f85a1a35d9c98d38742fabe03ff782970e2 WatchSource:0}: Error finding container 72fcae9f2c71a4ef3f5acafe3ae70f85a1a35d9c98d38742fabe03ff782970e2: Status 404 returned error can't find the container with id 72fcae9f2c71a4ef3f5acafe3ae70f85a1a35d9c98d38742fabe03ff782970e2 Jul 07 17:22:48.265194 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.265155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:48.265389 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.265232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:48.265389 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.265372 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:48.265389 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.265376 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:48.265550 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.265404 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:48.265550 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.265421 2579 projected.go:196] Error preparing data for projected volume kube-api-access-skt8n for pod openshift-network-diagnostics/network-check-target-f48zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:48.265550 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.265445 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs podName:0cba86e4-b5d0-446d-938b-ccf8dbc16c59 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:49.265424372 +0000 UTC m=+4.105738023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs") pod "network-metrics-daemon-xkrbk" (UID: "0cba86e4-b5d0-446d-938b-ccf8dbc16c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:48.265550 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.265468 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n podName:5298398f-faa0-40fb-b6c8-420e6a8c7fb3 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:49.265452801 +0000 UTC m=+4.105766455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-skt8n" (UniqueName: "kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n") pod "network-check-target-f48zt" (UID: "5298398f-faa0-40fb-b6c8-420e6a8c7fb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:48.366536 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.366501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:48.366751 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.366728 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:48.366831 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.366820 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret podName:de81cac2-c315-4825-927b-2b4c4f8f3790 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:49.366797863 +0000 UTC m=+4.207111520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret") pod "global-pull-secret-syncer-29cf4" (UID: "de81cac2-c315-4825-927b-2b4c4f8f3790") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:48.665426 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.665307 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:17:48 +0000 UTC" deadline="2028-04-14 23:11:42.27324952 +0000 UTC" Jul 07 17:22:48.665426 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.665369 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15533h48m53.607885201s" Jul 07 17:22:48.837396 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.836833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:48.837396 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:48.836973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:22:48.848274 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.848241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"72fcae9f2c71a4ef3f5acafe3ae70f85a1a35d9c98d38742fabe03ff782970e2"} Jul 07 17:22:48.850528 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.850479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" event={"ID":"74a13726-6b26-4315-baa7-68a633a5a363","Type":"ContainerStarted","Data":"5781644c5807d63479f1535ce9b45c9e40f73fc967e3c7efe5c77abf4464a1eb"} Jul 07 17:22:48.852373 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.852292 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ct9rk" event={"ID":"66c0087b-784a-460f-a00d-b2384a9eae45","Type":"ContainerStarted","Data":"aecc5e31a7db61638bc0601c795df3eb3880123a4dc22e9e1dfc7f348802dfe1"} Jul 07 17:22:48.853884 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.853809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttk6q" event={"ID":"509edf3c-5fd7-4e16-85b4-7405c98a2900","Type":"ContainerStarted","Data":"4d7f7bbf8749ecf92989e0b9fff1720c7312fc0331d53f514ba8f08c466d63f2"} Jul 07 17:22:48.856788 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.856716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqxtf" event={"ID":"06d0b994-7470-4e74-841d-4468df9db241","Type":"ContainerStarted","Data":"2552c2f5597c602f3a8a7ec09a6a1124009f1da17961489886f9097f48767a3e"} Jul 07 17:22:48.858861 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.858720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" event={"ID":"18fd592a-667c-4d50-8d14-c1fcacf2c2bc","Type":"ContainerStarted","Data":"7b0aaa12085a5eded978c7ab93d1a474cffc90e3dd82bf8e9dcc51608d1a6991"} Jul 07 17:22:48.860915 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.860835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fhpv8" event={"ID":"368511c4-3427-4707-9c1f-7e5d31d53fb4","Type":"ContainerStarted","Data":"22927f0a72b07809d2f820578bfc7eb3b58f815dbb921367443f104710c19037"} Jul 07 17:22:48.863851 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.863771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mw4xf" event={"ID":"955bef82-3b6a-47dd-9a19-113ab9aaebee","Type":"ContainerStarted","Data":"c4439568cb9b20bb5ca2612a6912314d5acc95735d07901824bc21aa496d1633"} Jul 07 17:22:48.867669 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:48.867603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerStarted","Data":"88721ccefd83e4af9536af7d13d6068531568316dc0b3857d28e074a5e18338a"} Jul 07 17:22:49.275258 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.275214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:49.275479 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.275315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:49.275479 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.275459 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:49.275656 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.275526 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs podName:0cba86e4-b5d0-446d-938b-ccf8dbc16c59 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:51.275506855 +0000 UTC m=+6.115820514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs") pod "network-metrics-daemon-xkrbk" (UID: "0cba86e4-b5d0-446d-938b-ccf8dbc16c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:49.275987 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.275965 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:49.276066 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.275996 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:49.276066 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.276009 2579 projected.go:196] Error preparing data for projected volume kube-api-access-skt8n for pod openshift-network-diagnostics/network-check-target-f48zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:49.276066 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.276056 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n podName:5298398f-faa0-40fb-b6c8-420e6a8c7fb3 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:51.276040891 +0000 UTC m=+6.116354546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-skt8n" (UniqueName: "kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n") pod "network-check-target-f48zt" (UID: "5298398f-faa0-40fb-b6c8-420e6a8c7fb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:49.375789 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.375747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:49.375976 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.375955 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:49.376057 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.376027 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret podName:de81cac2-c315-4825-927b-2b4c4f8f3790 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:51.376007592 +0000 UTC m=+6.216321248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret") pod "global-pull-secret-syncer-29cf4" (UID: "de81cac2-c315-4825-927b-2b4c4f8f3790") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:49.666215 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.666093 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:17:48 +0000 UTC" deadline="2028-01-09 06:41:44.066180964 +0000 UTC" Jul 07 17:22:49.666215 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.666137 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13213h18m54.400050019s" Jul 07 17:22:49.837015 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.836988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:49.837166 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.837121 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:22:49.839926 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.839896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:49.840062 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:49.840036 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:22:49.876366 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:49.876286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" event={"ID":"eff5104d7452b5ed4a3bf2742a0cf695","Type":"ContainerStarted","Data":"96e2b733c72aedc73e02fb1efde19108e854e18a42cf30f916033d68a7fe1b5c"} Jul 07 17:22:50.837502 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:50.836926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:50.837502 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:50.837067 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:22:50.891452 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:50.891364 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" event={"ID":"09305b88f1373ed8937b4f40e869c133","Type":"ContainerStarted","Data":"e81adcad81105b7ec6504f681d3cd519de8533b553808b7d2fcafdfd326fa159"} Jul 07 17:22:50.911818 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:50.910618 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" podStartSLOduration=3.91059616 podStartE2EDuration="3.91059616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:22:49.896415543 +0000 UTC m=+4.736729218" watchObservedRunningTime="2026-07-07 17:22:50.91059616 +0000 UTC m=+5.750909825" Jul 07 17:22:51.290046 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:51.289995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:51.290275 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:51.290256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:51.290497 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.290468 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:51.290497 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.290495 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:51.290643 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.290511 2579 projected.go:196] Error preparing data for projected volume kube-api-access-skt8n for pod openshift-network-diagnostics/network-check-target-f48zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:51.290643 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.290578 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n podName:5298398f-faa0-40fb-b6c8-420e6a8c7fb3 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:55.290556594 +0000 UTC m=+10.130870260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-skt8n" (UniqueName: "kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n") pod "network-check-target-f48zt" (UID: "5298398f-faa0-40fb-b6c8-420e6a8c7fb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:51.290776 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.290665 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:51.290776 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.290704 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs podName:0cba86e4-b5d0-446d-938b-ccf8dbc16c59 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:55.290691174 +0000 UTC m=+10.131004832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs") pod "network-metrics-daemon-xkrbk" (UID: "0cba86e4-b5d0-446d-938b-ccf8dbc16c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:51.392008 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:51.391350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:51.392008 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.391552 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:51.392008 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.391621 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret podName:de81cac2-c315-4825-927b-2b4c4f8f3790 nodeName:}" failed. No retries permitted until 2026-07-07 17:22:55.391601234 +0000 UTC m=+10.231914886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret") pod "global-pull-secret-syncer-29cf4" (UID: "de81cac2-c315-4825-927b-2b4c4f8f3790") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:51.838001 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:51.837469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:51.838001 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.837603 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:22:51.838001 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:51.837629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:51.838001 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:51.837848 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:22:52.837191 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:52.837115 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:52.837415 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:52.837258 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:22:53.839502 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:53.839469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:53.839997 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:53.839616 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:22:53.839997 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:53.839672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:53.839997 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:53.839813 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:22:53.899709 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:53.899676 2579 generic.go:356] "Generic (PLEG): container finished" podID="09305b88f1373ed8937b4f40e869c133" containerID="e81adcad81105b7ec6504f681d3cd519de8533b553808b7d2fcafdfd326fa159" exitCode=0 Jul 07 17:22:53.899852 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:53.899725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" event={"ID":"09305b88f1373ed8937b4f40e869c133","Type":"ContainerDied","Data":"e81adcad81105b7ec6504f681d3cd519de8533b553808b7d2fcafdfd326fa159"} Jul 07 17:22:54.837230 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:54.836707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:54.837230 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:54.836840 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:22:55.328423 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:55.328377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:55.328909 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:55.328453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:55.328909 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.328590 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:55.328909 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.328652 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs podName:0cba86e4-b5d0-446d-938b-ccf8dbc16c59 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:03.328632357 +0000 UTC m=+18.168946010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs") pod "network-metrics-daemon-xkrbk" (UID: "0cba86e4-b5d0-446d-938b-ccf8dbc16c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:22:55.328909 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.328746 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:22:55.328909 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.328764 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:22:55.328909 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.328777 2579 projected.go:196] Error preparing data for projected volume kube-api-access-skt8n for pod openshift-network-diagnostics/network-check-target-f48zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:55.328909 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.328815 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n podName:5298398f-faa0-40fb-b6c8-420e6a8c7fb3 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:03.328802887 +0000 UTC m=+18.169116538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-skt8n" (UniqueName: "kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n") pod "network-check-target-f48zt" (UID: "5298398f-faa0-40fb-b6c8-420e6a8c7fb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:22:55.429096 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:55.429061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:55.429240 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.429218 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:55.429314 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.429302 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret podName:de81cac2-c315-4825-927b-2b4c4f8f3790 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:03.429267396 +0000 UTC m=+18.269581047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret") pod "global-pull-secret-syncer-29cf4" (UID: "de81cac2-c315-4825-927b-2b4c4f8f3790") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:22:55.837943 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:55.837907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:55.838149 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.838038 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:22:55.838440 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:55.838416 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:55.838574 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:55.838487 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:22:56.837051 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:56.837018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:56.837511 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:56.837128 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:22:57.839570 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:57.839539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:57.840010 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:57.839664 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:22:57.840010 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:57.839724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:57.840010 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:57.839806 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:22:58.837117 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:58.837083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:22:58.837290 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:58.837217 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:22:59.837396 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:59.837363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:22:59.837396 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:22:59.837387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:22:59.837867 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:59.837492 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:22:59.837867 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:22:59.837635 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:00.836310 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:00.836282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:00.836473 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:00.836400 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:01.837662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:01.837055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:01.837662 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:01.837169 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:01.837662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:01.837526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:01.837662 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:01.837621 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:02.837207 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:02.837172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:02.837493 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:02.837304 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.388456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.388517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.388620 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.388621 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.388647 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.388660 2579 projected.go:196] Error preparing data for projected volume kube-api-access-skt8n for pod openshift-network-diagnostics/network-check-target-f48zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.388686 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs podName:0cba86e4-b5d0-446d-938b-ccf8dbc16c59 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:19.388669557 +0000 UTC m=+34.228983220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs") pod "network-metrics-daemon-xkrbk" (UID: "0cba86e4-b5d0-446d-938b-ccf8dbc16c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:23:03.388994 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.388710 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n podName:5298398f-faa0-40fb-b6c8-420e6a8c7fb3 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:19.388694587 +0000 UTC m=+34.229008259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-skt8n" (UniqueName: "kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n") pod "network-check-target-f48zt" (UID: "5298398f-faa0-40fb-b6c8-420e6a8c7fb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:23:03.490257 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.490009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:03.490374 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.490254 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:23:03.490374 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.490345 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret podName:de81cac2-c315-4825-927b-2b4c4f8f3790 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:19.490323278 +0000 UTC m=+34.330636927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret") pod "global-pull-secret-syncer-29cf4" (UID: "de81cac2-c315-4825-927b-2b4c4f8f3790") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:23:03.836809 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.836770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:03.837009 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.836893 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:03.837009 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.836925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:03.837009 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:03.836990 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:03.920681 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.920635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" event={"ID":"09305b88f1373ed8937b4f40e869c133","Type":"ContainerStarted","Data":"1e63bdf1a4efe33411be3b27d39a32fe8518abad8180598742413a0cd5cba46f"} Jul 07 17:23:03.922101 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.922074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" event={"ID":"74a13726-6b26-4315-baa7-68a633a5a363","Type":"ContainerStarted","Data":"24373550f650771b08fbe666bff41c7bc2386bf6c266449d7a2b7a58811f71ce"} Jul 07 17:23:03.923728 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.923709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ct9rk" event={"ID":"66c0087b-784a-460f-a00d-b2384a9eae45","Type":"ContainerStarted","Data":"7ac0cb1063c403d53cdf805bf54d2b866186491559031fb8f67d058dc6f784cc"} Jul 07 17:23:03.925055 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.925027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttk6q" event={"ID":"509edf3c-5fd7-4e16-85b4-7405c98a2900","Type":"ContainerStarted","Data":"6637ea7fd77b3b72b533fb9619b5affb62b8a7e327c27370802f785ef7fdbe21"} Jul 07 17:23:03.926431 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.926412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" event={"ID":"18fd592a-667c-4d50-8d14-c1fcacf2c2bc","Type":"ContainerStarted","Data":"3e7f3bd4c116d62bfbbf115920e0bb3192c3d9bb7df5e4034669ebbc33c6d229"} Jul 07 17:23:03.927718 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.927699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fhpv8" event={"ID":"368511c4-3427-4707-9c1f-7e5d31d53fb4","Type":"ContainerStarted","Data":"2059dfa3a940114594ef0663a147197e22bd0d3630db5819de64dc56971925d1"} Jul 07 17:23:03.928989 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.928967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mw4xf" event={"ID":"955bef82-3b6a-47dd-9a19-113ab9aaebee","Type":"ContainerStarted","Data":"a99d01bd2b3b49b4ded4102f5f59f6ce90a0e845de073ceee1e22cf64219ed89"} Jul 07 17:23:03.930388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.930367 2579 generic.go:356] "Generic (PLEG): container finished" podID="e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8" containerID="ae84691a7b735d16ccfebdb567e845a7dd1af8b7e92bcebfa2e93b24a81fdd07" exitCode=0 Jul 07 17:23:03.930472 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.930395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerDied","Data":"ae84691a7b735d16ccfebdb567e845a7dd1af8b7e92bcebfa2e93b24a81fdd07"} Jul 07 17:23:03.948688 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.948599 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" podStartSLOduration=16.948581993 podStartE2EDuration="16.948581993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:03.947728174 +0000 UTC m=+18.788041846" watchObservedRunningTime="2026-07-07 17:23:03.948581993 +0000 UTC m=+18.788895665" Jul 07 17:23:03.973116 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:03.973072 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mw4xf" podStartSLOduration=2.59640513 podStartE2EDuration="17.973058941s" totalImagesPullingTime="15.376653811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:03.97304109 +0000 UTC m=+18.813354761" watchObservedRunningTime="2026-07-07 17:23:03.973058941 +0000 UTC m=+18.813372611" Jul 07 17:23:04.003282 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:04.003232 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fhpv8" podStartSLOduration=2.629232148 podStartE2EDuration="18.003219922s" totalImagesPullingTime="15.373987774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.003116356 +0000 UTC m=+18.843430030" watchObservedRunningTime="2026-07-07 17:23:04.003219922 +0000 UTC m=+18.843533590" Jul 07 17:23:04.112544 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:04.112493 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ct9rk" podStartSLOduration=2.404289233 podStartE2EDuration="18.112475664s" totalImagesPullingTime="15.708186431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.112434644 +0000 UTC m=+18.952748315" watchObservedRunningTime="2026-07-07 17:23:04.112475664 +0000 UTC m=+18.952789359" Jul 07 17:23:04.159699 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:04.159657 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hrhfd" podStartSLOduration=2.765417142 podStartE2EDuration="18.159646275s" totalImagesPullingTime="15.394229133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.159478191 +0000 UTC m=+18.999791866" watchObservedRunningTime="2026-07-07 17:23:04.159646275 +0000 UTC m=+18.999959945" Jul 07 17:23:04.837256 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:04.837223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:04.837965 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:04.837359 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:04.935453 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:04.935383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqxtf" event={"ID":"06d0b994-7470-4e74-841d-4468df9db241","Type":"ContainerStarted","Data":"b0b8086d9cde098a0f017b2812eb539b970c5f832008f8981e5dddceb9493d97"} Jul 07 17:23:04.969755 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:04.969703 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ttk6q" podStartSLOduration=3.546613981 podStartE2EDuration="18.969684672s" totalImagesPullingTime="15.423070691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.206239879 +0000 UTC m=+19.046553551" watchObservedRunningTime="2026-07-07 17:23:04.969684672 +0000 UTC m=+19.809998344" Jul 07 17:23:04.970096 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:04.970067 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xqxtf" podStartSLOduration=3.542904729 podStartE2EDuration="18.970059937s" totalImagesPullingTime="15.427155208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:04.969560651 +0000 UTC m=+19.809874322" watchObservedRunningTime="2026-07-07 17:23:04.970059937 +0000 UTC m=+19.810373609" Jul 07 17:23:05.088001 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:05.087967 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 17:23:05.621445 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:05.621354 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T17:23:05.087985146Z","UUID":"8ad75968-ecfb-4ed9-a9d9-2ade51e76a33","Handler":null,"Name":"","Endpoint":""} Jul 07 17:23:05.622990 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:05.622967 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 17:23:05.623104 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:05.623008 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 17:23:05.838350 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:05.838091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:05.838753 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:05.838152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:05.838753 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:05.838459 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:05.838753 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:05.838546 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:05.939771 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:05.939688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" event={"ID":"18fd592a-667c-4d50-8d14-c1fcacf2c2bc","Type":"ContainerStarted","Data":"d05550707a1e2390bf850880e1d62c376df9cef02889020ed1884f03cb1c4836"} Jul 07 17:23:06.305211 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:06.305174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:23:06.305795 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:06.305774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:23:06.837165 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:06.837132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:06.837451 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:06.837284 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:06.942049 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:06.942008 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:23:06.942606 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:06.942586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ttk6q" Jul 07 17:23:07.388248 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:07.388214 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mw4xf_955bef82-3b6a-47dd-9a19-113ab9aaebee/dns-node-resolver/0.log" Jul 07 17:23:07.839475 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:07.838988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:07.839475 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:07.839121 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:07.839475 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:07.839176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:07.840089 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:07.839240 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:08.330329 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:08.330306 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fhpv8_368511c4-3427-4707-9c1f-7e5d31d53fb4/node-ca/0.log" Jul 07 17:23:08.836559 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:08.836524 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:08.836732 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:08.836636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:09.837089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:09.837057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:09.837506 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:09.837184 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:09.837506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:09.837222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:09.837506 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:09.837325 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:10.837115 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:10.837081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:10.837549 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:10.837234 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:11.836720 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:11.836686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:11.836909 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:11.836792 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:11.836909 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:11.836874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:11.836981 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:11.836966 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:12.836478 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.836301 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:12.836890 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:12.836543 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:12.955404 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.955376 2579 generic.go:356] "Generic (PLEG): container finished" podID="e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8" containerID="9991b363a554d8185e76115ea52d9ddfdc175976c8eda7ea06532f55f340a545" exitCode=0 Jul 07 17:23:12.955564 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.955473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerDied","Data":"9991b363a554d8185e76115ea52d9ddfdc175976c8eda7ea06532f55f340a545"} Jul 07 17:23:12.958473 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.958451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"bf325b167034016d1798769beea5bece1cac95bf1d890f0ef6d54517dd8488ca"} Jul 07 17:23:12.958584 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.958525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"c0f98f1126567500330869bb9cebdfeb48c8c922d9299827cff542c881dce295"} Jul 07 17:23:12.958584 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.958576 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"3f25b7e829f315bdb45562786269fe16469a4eadeb57ef87384b45561e4da5de"} Jul 07 17:23:12.958661 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.958611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"8f72fe72b96ec57fc34bc8a6817bbc89133e15fcd5f608b0cd3a93eca94570ab"} Jul 07 17:23:12.958661 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.958646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"ad8f371900525caaab50e5cd688650443100fa3dd35f1b4a38ae4f8cf5d1d932"} Jul 07 17:23:12.958749 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.958673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"9e6af7788359c8e3ebdcc862389601ac24be80f05ffeab256b50db08d119a0ee"} Jul 07 17:23:12.960647 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:12.960626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" event={"ID":"18fd592a-667c-4d50-8d14-c1fcacf2c2bc","Type":"ContainerStarted","Data":"55d1cd37310a2d8746a34d9f01daeb9f6559d5dcd54d22038b0e3db6276eb8cc"} Jul 07 17:23:13.009590 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:13.009551 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8qdp2" podStartSLOduration=3.1297761570000002 podStartE2EDuration="27.009538732s" totalImagesPullingTime="23.879762575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:13.009182095 +0000 UTC m=+27.849495766" watchObservedRunningTime="2026-07-07 17:23:13.009538732 +0000 UTC m=+27.849852403" Jul 07 17:23:13.836336 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:13.836310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:13.836517 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:13.836399 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:13.836517 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:13.836459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:13.836913 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:13.836576 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:14.836688 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:14.836660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:14.837045 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:14.836763 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:14.967234 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:14.967149 2579 generic.go:356] "Generic (PLEG): container finished" podID="e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8" containerID="3bf33d42a9b21bb16270832e8b77a856cb280aa6205d42a48ed1a597cac0cc60" exitCode=0 Jul 07 17:23:14.967402 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:14.967229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerDied","Data":"3bf33d42a9b21bb16270832e8b77a856cb280aa6205d42a48ed1a597cac0cc60"} Jul 07 17:23:14.970536 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:14.970517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"5f569153cca77a60ecbdac54076a4994e41651d6a90369e1f93933242a162b4b"} Jul 07 17:23:15.837154 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:15.837114 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:15.837730 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:15.837211 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:15.837730 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:15.837334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:15.837730 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:15.837456 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:16.836651 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:16.836620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:16.836835 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:16.836721 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:16.976872 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:16.976742 2579 generic.go:356] "Generic (PLEG): container finished" podID="e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8" containerID="62cdb05de0eb1f567e9d07cdad6807fd7e17a2ed80fc725353502be969fea6ab" exitCode=0 Jul 07 17:23:16.977282 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:16.976815 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerDied","Data":"62cdb05de0eb1f567e9d07cdad6807fd7e17a2ed80fc725353502be969fea6ab"} Jul 07 17:23:17.836762 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:17.836727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:17.836977 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:17.836881 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:17.836977 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:17.836950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:17.837049 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:17.837032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:17.982971 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:17.982929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" event={"ID":"35236574-483a-4f91-8738-d99b59016097","Type":"ContainerStarted","Data":"f225817b64514e2d9b376ddeac3edf20eabb694416e5760bbe3b16071fdc6213"} Jul 07 17:23:17.983438 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:17.983200 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:23:18.001011 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:18.000986 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:23:18.025990 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:18.025939 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" podStartSLOduration=8.070200657 podStartE2EDuration="32.025920938s" totalImagesPullingTime="23.955720281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:18.025413098 +0000 UTC m=+32.865726769" watchObservedRunningTime="2026-07-07 17:23:18.025920938 +0000 UTC m=+32.866234610" Jul 07 17:23:18.836795 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:18.836764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:18.837072 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:18.836883 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:18.985831 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:18.985802 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:23:18.986256 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:18.985896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:23:19.003680 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:19.003654 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:23:19.409690 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:19.409653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:19.409927 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:19.409731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:19.409927 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.409797 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:23:19.409927 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.409866 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs podName:0cba86e4-b5d0-446d-938b-ccf8dbc16c59 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:51.409848349 +0000 UTC m=+66.250162002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs") pod "network-metrics-daemon-xkrbk" (UID: "0cba86e4-b5d0-446d-938b-ccf8dbc16c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:23:19.409927 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.409877 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:23:19.409927 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.409894 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:23:19.409927 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.409905 2579 projected.go:196] Error preparing data for projected volume kube-api-access-skt8n for pod openshift-network-diagnostics/network-check-target-f48zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:23:19.410169 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.409954 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n podName:5298398f-faa0-40fb-b6c8-420e6a8c7fb3 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:51.409939977 +0000 UTC m=+66.250253647 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-skt8n" (UniqueName: "kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n") pod "network-check-target-f48zt" (UID: "5298398f-faa0-40fb-b6c8-420e6a8c7fb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:23:19.510100 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:19.510069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:19.510289 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.510194 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:23:19.510351 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.510293 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret podName:de81cac2-c315-4825-927b-2b4c4f8f3790 nodeName:}" failed. No retries permitted until 2026-07-07 17:23:51.510253305 +0000 UTC m=+66.350566959 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret") pod "global-pull-secret-syncer-29cf4" (UID: "de81cac2-c315-4825-927b-2b4c4f8f3790") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:23:19.839794 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:19.839759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:19.839794 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:19.839800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:19.839992 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.839887 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:19.840030 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:19.840001 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:20.836310 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:20.836286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:20.836788 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:20.836379 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:21.836450 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:21.836429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:21.836813 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:21.836567 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:21.836813 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:21.836611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:21.836813 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:21.836696 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:21.994080 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:21.994054 2579 generic.go:356] "Generic (PLEG): container finished" podID="e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8" containerID="38be9a3d1211f729460830a475342d302eaa6c13d72808b64d1e9b52877ef804" exitCode=0 Jul 07 17:23:21.994206 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:21.994109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerDied","Data":"38be9a3d1211f729460830a475342d302eaa6c13d72808b64d1e9b52877ef804"} Jul 07 17:23:22.836676 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:22.836643 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:22.837063 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:22.836764 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:22.998871 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:22.998839 2579 generic.go:356] "Generic (PLEG): container finished" podID="e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8" containerID="02704e2084fe9830753d4ecf61f5d0dc6a29fc51d277ef34880097f20e114c85" exitCode=0 Jul 07 17:23:22.999014 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:22.998904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerDied","Data":"02704e2084fe9830753d4ecf61f5d0dc6a29fc51d277ef34880097f20e114c85"} Jul 07 17:23:23.840146 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:23.840107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:23.840146 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:23.840138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:23.840698 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:23.840244 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:23.840698 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:23.840373 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:24.005328 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:24.005301 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7p99" event={"ID":"e87cb3ab-7292-4b2d-b9c4-d0416a7be8c8","Type":"ContainerStarted","Data":"61ba4fb060b16975d059f118db0f06dc5129c23b8e2f5366d014ca9e775055a3"} Jul 07 17:23:24.039387 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:24.039342 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d7p99" podStartSLOduration=7.9061918559999995 podStartE2EDuration="38.039327861s" totalImagesPullingTime="30.133136005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:24.038916495 +0000 UTC m=+38.879230166" watchObservedRunningTime="2026-07-07 17:23:24.039327861 +0000 UTC m=+38.879641533" Jul 07 17:23:24.837103 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:24.837073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:24.837286 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:24.837172 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:25.839530 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:25.839359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:25.839893 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:25.839360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:25.839893 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:25.839597 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:25.839893 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:25.839671 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:26.837143 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:26.837116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:26.837306 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:26.837210 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:27.836738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:27.836708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:27.837102 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:27.836708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:27.837102 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:27.836827 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:27.837102 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:27.836868 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:28.837113 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:28.837078 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:28.837555 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:28.837181 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:29.836953 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:29.836914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:29.836953 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:29.836951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:29.837476 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:29.837052 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:29.837476 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:29.837165 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:30.836397 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:30.836369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:30.836567 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:30.836465 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:31.837035 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:31.837008 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:31.837395 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:31.837104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:31.837395 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:31.837170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:31.837395 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:31.837288 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:32.836462 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:32.836438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:32.836632 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:32.836533 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:33.132155 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:33.132122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkrbk"] Jul 07 17:23:33.132739 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:33.132314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:33.132739 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:33.132405 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:33.134468 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:33.134444 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29cf4"] Jul 07 17:23:33.134566 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:33.134541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:33.134653 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:33.134632 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:33.135091 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:33.135073 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f48zt"] Jul 07 17:23:33.135169 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:33.135159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:33.135231 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:33.135214 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:34.837333 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:34.837304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:34.837849 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:34.837306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:34.837849 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:34.837407 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:34.837849 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:34.837306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:34.837849 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:34.837473 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:34.837849 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:34.837550 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:36.836468 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:36.836434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:36.836880 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:36.836536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:36.836880 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:36.836543 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkrbk" podUID="0cba86e4-b5d0-446d-938b-ccf8dbc16c59" Jul 07 17:23:36.836880 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:36.836545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:36.836880 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:36.836618 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29cf4" podUID="de81cac2-c315-4825-927b-2b4c4f8f3790" Jul 07 17:23:36.836880 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:36.836682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f48zt" podUID="5298398f-faa0-40fb-b6c8-420e6a8c7fb3" Jul 07 17:23:37.454754 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.454721 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeReady" Jul 07 17:23:37.454948 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.454838 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 17:23:37.508347 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.508318 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vsgtq"] Jul 07 17:23:37.510213 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.510198 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z25lv"] Jul 07 17:23:37.510356 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.510341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.512011 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.511984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.512718 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.512699 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 17:23:37.512718 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.512699 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pgv6p\"" Jul 07 17:23:37.512881 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.512733 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 17:23:37.514199 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.514184 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 17:23:37.514507 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.514494 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 17:23:37.514638 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.514624 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 17:23:37.514945 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.514930 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pcwpt\"" Jul 07 17:23:37.519956 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.519937 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z25lv"] Jul 07 17:23:37.523394 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.523372 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vsgtq"] Jul 07 17:23:37.532882 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.532864 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jtrzj"] Jul 07 17:23:37.534598 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.534585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.536732 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.536708 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k2gj6\"" Jul 07 17:23:37.536891 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.536862 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 17:23:37.537024 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.537007 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 17:23:37.537450 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.537426 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 17:23:37.538844 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.538821 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 17:23:37.549541 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.549521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jtrzj"] Jul 07 17:23:37.643691 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18b4b43e-4449-4593-8af1-eb189b662584-crio-socket\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.643691 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72ea041-019a-426c-8dd3-404f28a1f764-tmp-dir\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.643929 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72ea041-019a-426c-8dd3-404f28a1f764-metrics-tls\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.643929 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766dq\" (UniqueName: \"kubernetes.io/projected/f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518-kube-api-access-766dq\") pod \"ingress-canary-z25lv\" (UID: \"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518\") " pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.643929 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e72ea041-019a-426c-8dd3-404f28a1f764-config-volume\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.643929 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18b4b43e-4449-4593-8af1-eb189b662584-data-volume\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.643929 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b4b43e-4449-4593-8af1-eb189b662584-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.643929 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518-cert\") pod \"ingress-canary-z25lv\" (UID: \"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518\") " pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.644136 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27vd\" (UniqueName: \"kubernetes.io/projected/e72ea041-019a-426c-8dd3-404f28a1f764-kube-api-access-j27vd\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.644136 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.643996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9rh\" (UniqueName: \"kubernetes.io/projected/18b4b43e-4449-4593-8af1-eb189b662584-kube-api-access-tq9rh\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.644136 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.644014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b4b43e-4449-4593-8af1-eb189b662584-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.744311 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e72ea041-019a-426c-8dd3-404f28a1f764-config-volume\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.744487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18b4b43e-4449-4593-8af1-eb189b662584-data-volume\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.744487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b4b43e-4449-4593-8af1-eb189b662584-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.744487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518-cert\") pod \"ingress-canary-z25lv\" (UID: \"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518\") " pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.744487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27vd\" (UniqueName: \"kubernetes.io/projected/e72ea041-019a-426c-8dd3-404f28a1f764-kube-api-access-j27vd\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.744487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9rh\" (UniqueName: \"kubernetes.io/projected/18b4b43e-4449-4593-8af1-eb189b662584-kube-api-access-tq9rh\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.744487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b4b43e-4449-4593-8af1-eb189b662584-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.744773 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18b4b43e-4449-4593-8af1-eb189b662584-crio-socket\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.744773 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72ea041-019a-426c-8dd3-404f28a1f764-tmp-dir\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.744773 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72ea041-019a-426c-8dd3-404f28a1f764-metrics-tls\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.744773 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766dq\" (UniqueName: \"kubernetes.io/projected/f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518-kube-api-access-766dq\") pod \"ingress-canary-z25lv\" (UID: \"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518\") " pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.744773 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18b4b43e-4449-4593-8af1-eb189b662584-data-volume\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.745049 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.744986 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e72ea041-019a-426c-8dd3-404f28a1f764-config-volume\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.745115 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.745052 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b4b43e-4449-4593-8af1-eb189b662584-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.745115 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.745079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18b4b43e-4449-4593-8af1-eb189b662584-crio-socket\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.745354 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.745333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72ea041-019a-426c-8dd3-404f28a1f764-tmp-dir\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.748709 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.748687 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72ea041-019a-426c-8dd3-404f28a1f764-metrics-tls\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.748800 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.748692 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b4b43e-4449-4593-8af1-eb189b662584-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.748800 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.748731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518-cert\") pod \"ingress-canary-z25lv\" (UID: \"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518\") " pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.755490 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.755469 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27vd\" (UniqueName: \"kubernetes.io/projected/e72ea041-019a-426c-8dd3-404f28a1f764-kube-api-access-j27vd\") pod \"dns-default-vsgtq\" (UID: \"e72ea041-019a-426c-8dd3-404f28a1f764\") " pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.755917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.755900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9rh\" (UniqueName: \"kubernetes.io/projected/18b4b43e-4449-4593-8af1-eb189b662584-kube-api-access-tq9rh\") pod \"insights-runtime-extractor-jtrzj\" (UID: \"18b4b43e-4449-4593-8af1-eb189b662584\") " pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.756081 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.756060 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766dq\" (UniqueName: \"kubernetes.io/projected/f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518-kube-api-access-766dq\") pod \"ingress-canary-z25lv\" (UID: \"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518\") " pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.822498 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.822455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:37.828155 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.828137 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z25lv" Jul 07 17:23:37.846155 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.846134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jtrzj" Jul 07 17:23:37.987369 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.987339 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z25lv"] Jul 07 17:23:37.990177 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:37.990150 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5daea_5f8c_4e4e_8bfd_e4f21ea08518.slice/crio-30bdba8832640434273ae371ae8db407e1d1440c9c030b199d1157a136254bdc WatchSource:0}: Error finding container 30bdba8832640434273ae371ae8db407e1d1440c9c030b199d1157a136254bdc: Status 404 returned error can't find the container with id 30bdba8832640434273ae371ae8db407e1d1440c9c030b199d1157a136254bdc Jul 07 17:23:37.998014 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:37.997995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vsgtq"] Jul 07 17:23:38.000198 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:38.000167 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72ea041_019a_426c_8dd3_404f28a1f764.slice/crio-08b52ef7df83c7134bace54b8ef166b3d0e58bc37de5b47749b489e18fb321d1 WatchSource:0}: Error finding container 08b52ef7df83c7134bace54b8ef166b3d0e58bc37de5b47749b489e18fb321d1: Status 404 returned error can't find the container with id 08b52ef7df83c7134bace54b8ef166b3d0e58bc37de5b47749b489e18fb321d1 Jul 07 17:23:38.008355 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.008331 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jtrzj"] Jul 07 17:23:38.011380 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:38.011358 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b4b43e_4449_4593_8af1_eb189b662584.slice/crio-536331a1db7eac6d17b362c326d8d9fea5e320f4d70a50741f51a9958c3d8fa9 WatchSource:0}: Error finding container 536331a1db7eac6d17b362c326d8d9fea5e320f4d70a50741f51a9958c3d8fa9: Status 404 returned error can't find the container with id 536331a1db7eac6d17b362c326d8d9fea5e320f4d70a50741f51a9958c3d8fa9 Jul 07 17:23:38.033920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.033891 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtrzj" event={"ID":"18b4b43e-4449-4593-8af1-eb189b662584","Type":"ContainerStarted","Data":"536331a1db7eac6d17b362c326d8d9fea5e320f4d70a50741f51a9958c3d8fa9"} Jul 07 17:23:38.034975 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.034952 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z25lv" event={"ID":"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518","Type":"ContainerStarted","Data":"30bdba8832640434273ae371ae8db407e1d1440c9c030b199d1157a136254bdc"} Jul 07 17:23:38.035865 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.035846 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsgtq" event={"ID":"e72ea041-019a-426c-8dd3-404f28a1f764","Type":"ContainerStarted","Data":"08b52ef7df83c7134bace54b8ef166b3d0e58bc37de5b47749b489e18fb321d1"} Jul 07 17:23:38.837325 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.837290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:38.837778 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.837747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:38.838205 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.838162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:38.841033 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.841010 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:23:38.842557 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.842535 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 17:23:38.843820 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.843799 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vmhtn\"" Jul 07 17:23:38.844292 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.844254 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7rcfs\"" Jul 07 17:23:38.844386 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.844296 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 17:23:38.844386 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:38.844254 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:23:39.878991 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.878132 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c59fbdcc4-bjs6m"] Jul 07 17:23:39.881613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.880302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:39.886878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.886854 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 17:23:39.887087 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.887071 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 17:23:39.887240 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.887223 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 17:23:39.887620 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.887481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 17:23:39.888058 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.888040 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zm2j4\"" Jul 07 17:23:39.891795 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.891775 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 17:23:39.892563 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.892547 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 17:23:39.893764 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.893748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 17:23:39.896303 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.896102 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 17:23:39.903344 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:39.903321 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c59fbdcc4-bjs6m"] Jul 07 17:23:40.060287 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.060236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-oauth-config\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.060287 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.060279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-console-config\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.060494 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.060343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-trusted-ca-bundle\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.060494 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.060385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-service-ca\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.060580 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.060499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7h2z\" (UniqueName: \"kubernetes.io/projected/851db565-01a9-4ef5-be99-3853ce29fd2a-kube-api-access-b7h2z\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.060580 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.060538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-oauth-serving-cert\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.060580 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.060566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-serving-cert\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.160950 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.160868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-oauth-config\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.160950 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.160912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-console-config\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.160950 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.160941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-trusted-ca-bundle\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.161199 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.160967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-service-ca\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.161199 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.161038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7h2z\" (UniqueName: \"kubernetes.io/projected/851db565-01a9-4ef5-be99-3853ce29fd2a-kube-api-access-b7h2z\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.161199 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.161065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-oauth-serving-cert\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.161199 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.161089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-serving-cert\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.163109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.162507 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-trusted-ca-bundle\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.163109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.162644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-service-ca\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.163109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.162939 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-console-config\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.163109 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.163068 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-oauth-serving-cert\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.164371 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.164351 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-serving-cert\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.164736 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.164717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-oauth-config\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.171528 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.171484 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7h2z\" (UniqueName: \"kubernetes.io/projected/851db565-01a9-4ef5-be99-3853ce29fd2a-kube-api-access-b7h2z\") pod \"console-c59fbdcc4-bjs6m\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:40.195256 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:40.195229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:43.600461 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:43.600436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c59fbdcc4-bjs6m"] Jul 07 17:23:43.602881 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:43.602860 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851db565_01a9_4ef5_be99_3853ce29fd2a.slice/crio-52e322ba2a8f9480a04abfec41910ed12b44f616bb758d4939a41b39b49a0c9a WatchSource:0}: Error finding container 52e322ba2a8f9480a04abfec41910ed12b44f616bb758d4939a41b39b49a0c9a: Status 404 returned error can't find the container with id 52e322ba2a8f9480a04abfec41910ed12b44f616bb758d4939a41b39b49a0c9a Jul 07 17:23:44.052440 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:44.052371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z25lv" event={"ID":"f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518","Type":"ContainerStarted","Data":"611f32a54f742a7c10536eda3d53f77d348b8916219e4b3628f9982609ebfb62"} Jul 07 17:23:44.054400 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:44.054380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtrzj" event={"ID":"18b4b43e-4449-4593-8af1-eb189b662584","Type":"ContainerStarted","Data":"98ad7e5f7ad150b87a6c7fc463778c3914b2f9c7255c51b47a38b6b29c9b119c"} Jul 07 17:23:44.056159 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:44.056130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsgtq" event={"ID":"e72ea041-019a-426c-8dd3-404f28a1f764","Type":"ContainerStarted","Data":"9232dcf1bfbd551920a3d34b63eaf8e56f64319b9fa84692c6bb2de2d38ba2e9"} Jul 07 17:23:44.056257 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:44.056169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsgtq" event={"ID":"e72ea041-019a-426c-8dd3-404f28a1f764","Type":"ContainerStarted","Data":"863515fb43fc02c2391e9feb67ecb941d87cefe5c462af7c6234cbb7b92874cd"} Jul 07 17:23:44.056257 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:44.056239 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:44.057290 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:44.057250 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59fbdcc4-bjs6m" event={"ID":"851db565-01a9-4ef5-be99-3853ce29fd2a","Type":"ContainerStarted","Data":"52e322ba2a8f9480a04abfec41910ed12b44f616bb758d4939a41b39b49a0c9a"} Jul 07 17:23:44.080341 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:44.080303 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z25lv" podStartSLOduration=1.605977276 podStartE2EDuration="7.080288663s" totalImagesPullingTime="5.474311387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:44.07987818 +0000 UTC m=+58.920191851" watchObservedRunningTime="2026-07-07 17:23:44.080288663 +0000 UTC m=+58.920602325" Jul 07 17:23:45.061436 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:45.061325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtrzj" event={"ID":"18b4b43e-4449-4593-8af1-eb189b662584","Type":"ContainerStarted","Data":"3aa25817d99e062a2676e106017755c9524be69c818f82754bc584d12e64edfb"} Jul 07 17:23:47.779020 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.778974 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vsgtq" podStartSLOduration=5.315283021 podStartE2EDuration="10.778956927s" totalImagesPullingTime="5.463673906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:44.111485523 +0000 UTC m=+58.951799195" watchObservedRunningTime="2026-07-07 17:23:47.778956927 +0000 UTC m=+62.619270599" Jul 07 17:23:47.779828 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.779808 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs"] Jul 07 17:23:47.782312 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.782286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:47.785910 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.785892 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 17:23:47.786878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.786218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4fjbj\"" Jul 07 17:23:47.786878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.786322 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 17:23:47.786878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.786466 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 17:23:47.786878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.786701 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 17:23:47.786878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.786715 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 17:23:47.805538 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.805521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs"] Jul 07 17:23:47.806140 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.806123 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb"] Jul 07 17:23:47.808916 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.808892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:47.812003 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.811781 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-klqqn\"" Jul 07 17:23:47.812003 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.811813 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 17:23:47.812003 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.811901 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 17:23:47.812003 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.811997 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 17:23:47.820554 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.820533 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb"] Jul 07 17:23:47.825551 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.825522 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tnjpm"] Jul 07 17:23:47.827323 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.827309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.829686 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.829668 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 17:23:47.829783 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.829760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-chml4\"" Jul 07 17:23:47.830206 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.830178 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 17:23:47.830597 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.830580 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 17:23:47.922696 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:47.922837 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-textfile\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.922837 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsr4s\" (UniqueName: \"kubernetes.io/projected/7ba304d4-350f-4117-847c-c3ec3fd08f78-kube-api-access-tsr4s\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.922837 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.922943 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.922943 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:47.922943 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-root\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.922943 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:47.923079 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.922955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-tls\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.923079 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:47.923079 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de194883-e38b-41b6-b9ec-aec7ffe6342b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:47.923184 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de194883-e38b-41b6-b9ec-aec7ffe6342b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:47.923184 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-wtmp\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.923276 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9p8d\" (UniqueName: \"kubernetes.io/projected/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-api-access-w9p8d\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:47.923276 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-sys\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.923276 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmw9\" (UniqueName: \"kubernetes.io/projected/de194883-e38b-41b6-b9ec-aec7ffe6342b-kube-api-access-hlmw9\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:47.923399 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de194883-e38b-41b6-b9ec-aec7ffe6342b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:47.923399 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ba304d4-350f-4117-847c-c3ec3fd08f78-metrics-client-ca\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:47.923399 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:47.923338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.024578 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.024578 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-root\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.024746 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.024746 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-tls\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.024746 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024642 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-root\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.024746 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:48.024668 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 17:23:48.024746 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.024746 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:23:48.024727 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-tls podName:4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f nodeName:}" failed. No retries permitted until 2026-07-07 17:23:48.524709028 +0000 UTC m=+63.365022684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-rjwjb" (UID: "4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f") : secret "kube-state-metrics-tls" not found Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de194883-e38b-41b6-b9ec-aec7ffe6342b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de194883-e38b-41b6-b9ec-aec7ffe6342b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-wtmp\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9p8d\" (UniqueName: \"kubernetes.io/projected/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-api-access-w9p8d\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-sys\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024926 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmw9\" (UniqueName: \"kubernetes.io/projected/de194883-e38b-41b6-b9ec-aec7ffe6342b-kube-api-access-hlmw9\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.025004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.024984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de194883-e38b-41b6-b9ec-aec7ffe6342b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ba304d4-350f-4117-847c-c3ec3fd08f78-metrics-client-ca\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-wtmp\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-textfile\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025134 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ba304d4-350f-4117-847c-c3ec3fd08f78-sys\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr4s\" (UniqueName: \"kubernetes.io/projected/7ba304d4-350f-4117-847c-c3ec3fd08f78-kube-api-access-tsr4s\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.025841 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.025826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-textfile\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.026141 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.026112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.026227 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.026202 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.026424 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.026401 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.026652 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.026626 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ba304d4-350f-4117-847c-c3ec3fd08f78-metrics-client-ca\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.026734 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.026686 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de194883-e38b-41b6-b9ec-aec7ffe6342b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.027476 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.027451 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-tls\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.027684 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.027661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.027761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.027727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de194883-e38b-41b6-b9ec-aec7ffe6342b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.028100 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.028085 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ba304d4-350f-4117-847c-c3ec3fd08f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.028280 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.028253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de194883-e38b-41b6-b9ec-aec7ffe6342b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.034584 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.034558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmw9\" (UniqueName: \"kubernetes.io/projected/de194883-e38b-41b6-b9ec-aec7ffe6342b-kube-api-access-hlmw9\") pod \"openshift-state-metrics-54cf96dfcf-9nlxs\" (UID: \"de194883-e38b-41b6-b9ec-aec7ffe6342b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.035180 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.035158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsr4s\" (UniqueName: \"kubernetes.io/projected/7ba304d4-350f-4117-847c-c3ec3fd08f78-kube-api-access-tsr4s\") pod \"node-exporter-tnjpm\" (UID: \"7ba304d4-350f-4117-847c-c3ec3fd08f78\") " pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.036201 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.036171 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9p8d\" (UniqueName: \"kubernetes.io/projected/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-api-access-w9p8d\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.074466 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.074443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtrzj" event={"ID":"18b4b43e-4449-4593-8af1-eb189b662584","Type":"ContainerStarted","Data":"7c81bfa1e25dfba9a743d2cbd7976e62d1003ea43f5be1c13a83464693299c93"} Jul 07 17:23:48.075985 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.075956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59fbdcc4-bjs6m" event={"ID":"851db565-01a9-4ef5-be99-3853ce29fd2a","Type":"ContainerStarted","Data":"dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13"} Jul 07 17:23:48.099890 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.099841 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jtrzj" podStartSLOduration=1.5862859 podStartE2EDuration="11.099825842s" totalImagesPullingTime="9.513539942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:23:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:48.09910286 +0000 UTC m=+62.939416530" watchObservedRunningTime="2026-07-07 17:23:48.099825842 +0000 UTC m=+62.940139516" Jul 07 17:23:48.118417 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.118399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" Jul 07 17:23:48.123786 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.123746 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c59fbdcc4-bjs6m" podStartSLOduration=4.961036708 podStartE2EDuration="9.123734575s" totalImagesPullingTime="4.162697867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:48.122853714 +0000 UTC m=+62.963167396" watchObservedRunningTime="2026-07-07 17:23:48.123734575 +0000 UTC m=+62.964048246" Jul 07 17:23:48.142729 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.142706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tnjpm" Jul 07 17:23:48.151480 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:48.151461 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba304d4_350f_4117_847c_c3ec3fd08f78.slice/crio-db988b965ebd4af9d2dd2a987c726d4b41fa4f893d44182c78e883f61bcf7ea4 WatchSource:0}: Error finding container db988b965ebd4af9d2dd2a987c726d4b41fa4f893d44182c78e883f61bcf7ea4: Status 404 returned error can't find the container with id db988b965ebd4af9d2dd2a987c726d4b41fa4f893d44182c78e883f61bcf7ea4 Jul 07 17:23:48.244497 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.244472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs"] Jul 07 17:23:48.247713 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:48.247691 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde194883_e38b_41b6_b9ec_aec7ffe6342b.slice/crio-99de939875251b30cb0f8426fdd79ee6ad33cadb965d2b523e247691ef8bcb49 WatchSource:0}: Error finding container 99de939875251b30cb0f8426fdd79ee6ad33cadb965d2b523e247691ef8bcb49: Status 404 returned error can't find the container with id 99de939875251b30cb0f8426fdd79ee6ad33cadb965d2b523e247691ef8bcb49 Jul 07 17:23:48.529964 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.529878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.532242 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.532220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-rjwjb\" (UID: \"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.736949 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.736920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" Jul 07 17:23:48.872442 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:48.872409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb"] Jul 07 17:23:48.876563 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:48.876536 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e46accb_4fa2_4626_bf00_0c8b9b4c5b6f.slice/crio-3412acce7a4f079856471547a6c2bdc7dfea2a93e85fa3c0b68ff5681cf43104 WatchSource:0}: Error finding container 3412acce7a4f079856471547a6c2bdc7dfea2a93e85fa3c0b68ff5681cf43104: Status 404 returned error can't find the container with id 3412acce7a4f079856471547a6c2bdc7dfea2a93e85fa3c0b68ff5681cf43104 Jul 07 17:23:49.081136 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:49.081042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnjpm" event={"ID":"7ba304d4-350f-4117-847c-c3ec3fd08f78","Type":"ContainerStarted","Data":"db988b965ebd4af9d2dd2a987c726d4b41fa4f893d44182c78e883f61bcf7ea4"} Jul 07 17:23:49.087131 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:49.087099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" event={"ID":"de194883-e38b-41b6-b9ec-aec7ffe6342b","Type":"ContainerStarted","Data":"190b7de8d4a8ece4d64b49a985402e14b70537cf368b0fa7ae26b28ed1b0793b"} Jul 07 17:23:49.087260 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:49.087174 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" event={"ID":"de194883-e38b-41b6-b9ec-aec7ffe6342b","Type":"ContainerStarted","Data":"1f688061c484d4771b858af07a60a0a3b319712e535c09fad7eeaea88091d8e5"} Jul 07 17:23:49.087260 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:49.087195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" event={"ID":"de194883-e38b-41b6-b9ec-aec7ffe6342b","Type":"ContainerStarted","Data":"99de939875251b30cb0f8426fdd79ee6ad33cadb965d2b523e247691ef8bcb49"} Jul 07 17:23:49.088851 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:49.088754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" event={"ID":"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f","Type":"ContainerStarted","Data":"3412acce7a4f079856471547a6c2bdc7dfea2a93e85fa3c0b68ff5681cf43104"} Jul 07 17:23:50.096171 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.096138 2579 generic.go:356] "Generic (PLEG): container finished" podID="7ba304d4-350f-4117-847c-c3ec3fd08f78" containerID="2c699129dcbc7b1fc56541904617a4a222708335170a84c8e220ef538a7e27d3" exitCode=0 Jul 07 17:23:50.096607 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.096223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnjpm" event={"ID":"7ba304d4-350f-4117-847c-c3ec3fd08f78","Type":"ContainerDied","Data":"2c699129dcbc7b1fc56541904617a4a222708335170a84c8e220ef538a7e27d3"} Jul 07 17:23:50.098443 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.098420 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" event={"ID":"de194883-e38b-41b6-b9ec-aec7ffe6342b","Type":"ContainerStarted","Data":"214b480295e75ae0de3452f3c9bf464d6f0ce6d27fda09200530b300a35b81d2"} Jul 07 17:23:50.195720 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.195684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:50.195819 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.195759 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:50.200768 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.200746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:50.226893 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.226837 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9nlxs" podStartSLOduration=1.845453333 podStartE2EDuration="3.226816037s" totalImagesPullingTime="1.381362704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:50.144257126 +0000 UTC m=+64.984570798" watchObservedRunningTime="2026-07-07 17:23:50.226816037 +0000 UTC m=+65.067129708" Jul 07 17:23:50.808334 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.808298 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-789dd4f6c8-7lphr"] Jul 07 17:23:50.811583 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.811555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.819805 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.819785 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 17:23:50.819935 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.819806 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 17:23:50.820042 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.820029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5skmto3ht1qdm\"" Jul 07 17:23:50.820120 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.820106 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 17:23:50.820208 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.820182 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 17:23:50.822836 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.822816 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 17:23:50.823122 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.823105 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zf2st\"" Jul 07 17:23:50.845666 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.845644 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-789dd4f6c8-7lphr"] Jul 07 17:23:50.949769 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.949745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-tls\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.949862 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.949792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.949862 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.949821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.949862 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.949848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.950046 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.949904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-grpc-tls\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.950046 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.949941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.950046 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.949975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-metrics-client-ca\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:50.950046 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:50.950000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7vxk\" (UniqueName: \"kubernetes.io/projected/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-kube-api-access-q7vxk\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.006230 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.006210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4bsf6" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-tls\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-grpc-tls\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-metrics-client-ca\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.050744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7vxk\" (UniqueName: \"kubernetes.io/projected/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-kube-api-access-q7vxk\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.051995 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.051661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-metrics-client-ca\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.055144 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.055116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.055284 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.055219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-tls\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.055658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.055606 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.055843 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.055818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-grpc-tls\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.057291 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.057224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.057700 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.057680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.088310 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.088182 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7vxk\" (UniqueName: \"kubernetes.io/projected/f4b1a2ef-8141-449c-99bf-0a2bd48b1ade-kube-api-access-q7vxk\") pod \"thanos-querier-789dd4f6c8-7lphr\" (UID: \"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade\") " pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.103547 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.103519 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" event={"ID":"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f","Type":"ContainerStarted","Data":"abe9c4397ce06ace4be78ecf52820a950dfa95bdd43a7bf61927cbb5a8ddde8d"} Jul 07 17:23:51.103878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.103571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" event={"ID":"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f","Type":"ContainerStarted","Data":"39c1bcbae88fd9e2a2d7e892c0475ba374e483f03c6fa63ccac4f1084575330d"} Jul 07 17:23:51.105657 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.105624 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnjpm" event={"ID":"7ba304d4-350f-4117-847c-c3ec3fd08f78","Type":"ContainerStarted","Data":"10fcb8d7735bdf796e92f401c4c9535a316fa704dba39e1ade60402ad98d482c"} Jul 07 17:23:51.105777 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.105688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnjpm" event={"ID":"7ba304d4-350f-4117-847c-c3ec3fd08f78","Type":"ContainerStarted","Data":"416e7859e2f2280a155bccc11f006f1c1db3fde3ad6e86188d0d76f09f01090a"} Jul 07 17:23:51.111492 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.111468 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:23:51.122275 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.122241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:23:51.139623 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.139583 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tnjpm" podStartSLOduration=2.486207441 podStartE2EDuration="4.139568903s" totalImagesPullingTime="1.572685251s" totalInitContainerRuntime="80.676211ms" isStatefulPod=true podCreationTimestamp="2026-07-07 17:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:51.138505446 +0000 UTC m=+65.978819118" watchObservedRunningTime="2026-07-07 17:23:51.139568903 +0000 UTC m=+65.979882573" Jul 07 17:23:51.282034 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.282002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-789dd4f6c8-7lphr"] Jul 07 17:23:51.285432 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:51.285409 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b1a2ef_8141_449c_99bf_0a2bd48b1ade.slice/crio-12318242573abf827c267203dc93dd0ef8ddb33738a20d44e8c6f3de132d4ee1 WatchSource:0}: Error finding container 12318242573abf827c267203dc93dd0ef8ddb33738a20d44e8c6f3de132d4ee1: Status 404 returned error can't find the container with id 12318242573abf827c267203dc93dd0ef8ddb33738a20d44e8c6f3de132d4ee1 Jul 07 17:23:51.454622 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.454542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:51.454622 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.454603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:51.457531 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.457512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 17:23:51.457623 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.457512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:23:51.467885 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.467862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cba86e4-b5d0-446d-938b-ccf8dbc16c59-metrics-certs\") pod \"network-metrics-daemon-xkrbk\" (UID: \"0cba86e4-b5d0-446d-938b-ccf8dbc16c59\") " pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:51.468183 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.468171 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 17:23:51.478746 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.478723 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skt8n\" (UniqueName: \"kubernetes.io/projected/5298398f-faa0-40fb-b6c8-420e6a8c7fb3-kube-api-access-skt8n\") pod \"network-check-target-f48zt\" (UID: \"5298398f-faa0-40fb-b6c8-420e6a8c7fb3\") " pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:51.555834 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.555798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:51.558365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.558350 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:23:51.568514 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.568491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de81cac2-c315-4825-927b-2b4c4f8f3790-original-pull-secret\") pod \"global-pull-secret-syncer-29cf4\" (UID: \"de81cac2-c315-4825-927b-2b4c4f8f3790\") " pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:51.754929 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.754908 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7rcfs\"" Jul 07 17:23:51.763118 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.763095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkrbk" Jul 07 17:23:51.763206 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.763140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29cf4" Jul 07 17:23:51.776124 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.776101 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vmhtn\"" Jul 07 17:23:51.782158 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.782129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:51.913775 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.913701 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkrbk"] Jul 07 17:23:51.917718 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:51.917693 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cba86e4_b5d0_446d_938b_ccf8dbc16c59.slice/crio-1af32f994e5471c92cf97a5e7e2ab08b55b0cc77a7906c3c6573809c7963fa69 WatchSource:0}: Error finding container 1af32f994e5471c92cf97a5e7e2ab08b55b0cc77a7906c3c6573809c7963fa69: Status 404 returned error can't find the container with id 1af32f994e5471c92cf97a5e7e2ab08b55b0cc77a7906c3c6573809c7963fa69 Jul 07 17:23:51.929749 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.929725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29cf4"] Jul 07 17:23:51.932647 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:51.932620 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde81cac2_c315_4825_927b_2b4c4f8f3790.slice/crio-380dece18a34ca34b01a3908b2a14a58bb38205d91f414da140a1267f491fa20 WatchSource:0}: Error finding container 380dece18a34ca34b01a3908b2a14a58bb38205d91f414da140a1267f491fa20: Status 404 returned error can't find the container with id 380dece18a34ca34b01a3908b2a14a58bb38205d91f414da140a1267f491fa20 Jul 07 17:23:51.947697 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:51.947666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f48zt"] Jul 07 17:23:51.950188 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:51.950143 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5298398f_faa0_40fb_b6c8_420e6a8c7fb3.slice/crio-83889ff5d52cef1b51956ee6b89b2250455af3e59fe04e1a3711564826aab963 WatchSource:0}: Error finding container 83889ff5d52cef1b51956ee6b89b2250455af3e59fe04e1a3711564826aab963: Status 404 returned error can't find the container with id 83889ff5d52cef1b51956ee6b89b2250455af3e59fe04e1a3711564826aab963 Jul 07 17:23:52.111363 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.111290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29cf4" event={"ID":"de81cac2-c315-4825-927b-2b4c4f8f3790","Type":"ContainerStarted","Data":"380dece18a34ca34b01a3908b2a14a58bb38205d91f414da140a1267f491fa20"} Jul 07 17:23:52.112630 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.112605 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkrbk" event={"ID":"0cba86e4-b5d0-446d-938b-ccf8dbc16c59","Type":"ContainerStarted","Data":"1af32f994e5471c92cf97a5e7e2ab08b55b0cc77a7906c3c6573809c7963fa69"} Jul 07 17:23:52.115152 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.115126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" event={"ID":"4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f","Type":"ContainerStarted","Data":"51b988da37c550de732e0b2b13ee2dcde66611e33b6fe97f67f0479dccff820a"} Jul 07 17:23:52.116517 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.116495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f48zt" event={"ID":"5298398f-faa0-40fb-b6c8-420e6a8c7fb3","Type":"ContainerStarted","Data":"83889ff5d52cef1b51956ee6b89b2250455af3e59fe04e1a3711564826aab963"} Jul 07 17:23:52.117733 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.117712 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" event={"ID":"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade","Type":"ContainerStarted","Data":"12318242573abf827c267203dc93dd0ef8ddb33738a20d44e8c6f3de132d4ee1"} Jul 07 17:23:52.136378 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.136333 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rjwjb" podStartSLOduration=3.091960022 podStartE2EDuration="5.136319069s" totalImagesPullingTime="2.044359047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:52.13543118 +0000 UTC m=+66.975744856" watchObservedRunningTime="2026-07-07 17:23:52.136319069 +0000 UTC m=+66.976632739" Jul 07 17:23:52.572090 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.570595 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql"] Jul 07 17:23:52.596955 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.596409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql"] Jul 07 17:23:52.596955 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.596555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:52.600143 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.600117 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v5grp\"" Jul 07 17:23:52.600447 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.600423 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 17:23:52.680426 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.680390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6zq\" (UniqueName: \"kubernetes.io/projected/c023f6ac-d192-4c18-8d9e-04d029798295-kube-api-access-zq6zq\") pod \"monitoring-plugin-d5dcff7cf-cxgql\" (UID: \"c023f6ac-d192-4c18-8d9e-04d029798295\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:52.680592 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.680483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c023f6ac-d192-4c18-8d9e-04d029798295-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-cxgql\" (UID: \"c023f6ac-d192-4c18-8d9e-04d029798295\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:52.782007 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.781204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6zq\" (UniqueName: \"kubernetes.io/projected/c023f6ac-d192-4c18-8d9e-04d029798295-kube-api-access-zq6zq\") pod \"monitoring-plugin-d5dcff7cf-cxgql\" (UID: \"c023f6ac-d192-4c18-8d9e-04d029798295\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:52.782007 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.781295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c023f6ac-d192-4c18-8d9e-04d029798295-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-cxgql\" (UID: \"c023f6ac-d192-4c18-8d9e-04d029798295\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:52.799720 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.799691 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c023f6ac-d192-4c18-8d9e-04d029798295-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-cxgql\" (UID: \"c023f6ac-d192-4c18-8d9e-04d029798295\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:52.811599 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.811565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6zq\" (UniqueName: \"kubernetes.io/projected/c023f6ac-d192-4c18-8d9e-04d029798295-kube-api-access-zq6zq\") pod \"monitoring-plugin-d5dcff7cf-cxgql\" (UID: \"c023f6ac-d192-4c18-8d9e-04d029798295\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:52.908614 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:52.908541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:53.028079 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.028042 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57fccdd65f-r8qmc"] Jul 07 17:23:53.045990 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.045964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.049588 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.049375 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9ddng\"" Jul 07 17:23:53.049588 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.049396 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 17:23:53.049588 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.049448 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 17:23:53.049588 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.049496 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 17:23:53.049859 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.049640 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 17:23:53.049920 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.049858 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 17:23:53.050614 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.050593 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57fccdd65f-r8qmc"] Jul 07 17:23:53.059951 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.059929 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 17:23:53.083293 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-secret-telemeter-client\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.083476 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.083591 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-telemeter-client-tls\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.083738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.083738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-federate-client-tls\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.083738 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-metrics-client-ca\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.084105 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z29d\" (UniqueName: \"kubernetes.io/projected/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-kube-api-access-8z29d\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.084105 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.083792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-serving-certs-ca-bundle\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-telemeter-client-tls\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-federate-client-tls\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-metrics-client-ca\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z29d\" (UniqueName: \"kubernetes.io/projected/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-kube-api-access-8z29d\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-serving-certs-ca-bundle\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-secret-telemeter-client\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.184613 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.184520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.189257 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.186741 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.189257 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.187701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-serving-certs-ca-bundle\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.189257 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.188970 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-metrics-client-ca\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.193055 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.193005 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.198958 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.198586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-telemeter-client-tls\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.199629 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.199564 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-federate-client-tls\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.201546 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.200146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z29d\" (UniqueName: \"kubernetes.io/projected/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-kube-api-access-8z29d\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.201890 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.201849 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4514d7be-35de-4eac-b3c1-0b9b3a41f9bf-secret-telemeter-client\") pod \"telemeter-client-57fccdd65f-r8qmc\" (UID: \"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf\") " pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.237251 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.237225 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql"] Jul 07 17:23:53.243734 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:53.243705 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc023f6ac_d192_4c18_8d9e_04d029798295.slice/crio-4fd41228209e279502fdc21fbc887613f7568c5198da8b51ddd39e9f3fc17bea WatchSource:0}: Error finding container 4fd41228209e279502fdc21fbc887613f7568c5198da8b51ddd39e9f3fc17bea: Status 404 returned error can't find the container with id 4fd41228209e279502fdc21fbc887613f7568c5198da8b51ddd39e9f3fc17bea Jul 07 17:23:53.362428 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.362401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" Jul 07 17:23:53.641299 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.641257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57fccdd65f-r8qmc"] Jul 07 17:23:53.642396 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:53.642364 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4514d7be_35de_4eac_b3c1_0b9b3a41f9bf.slice/crio-42a5163c84c822a596cb0ca28d1f9bfec2a8d113794a16d1c5988620d1d4889b WatchSource:0}: Error finding container 42a5163c84c822a596cb0ca28d1f9bfec2a8d113794a16d1c5988620d1d4889b: Status 404 returned error can't find the container with id 42a5163c84c822a596cb0ca28d1f9bfec2a8d113794a16d1c5988620d1d4889b Jul 07 17:23:53.796516 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.796492 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77ddd94b9d-68nkx"] Jul 07 17:23:53.799327 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.799298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.814983 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.814959 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77ddd94b9d-68nkx"] Jul 07 17:23:53.889660 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.889628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-oauth-config\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.889810 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.889674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzcsr\" (UniqueName: \"kubernetes.io/projected/806c066d-d8dd-4b4b-aad5-23336f9e1248-kube-api-access-gzcsr\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.889810 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.889705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-oauth-serving-cert\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.889810 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.889749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-config\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.889810 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.889784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-serving-cert\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.889991 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.889811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-trusted-ca-bundle\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.889991 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.889865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-service-ca\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.990793 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.990758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-service-ca\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.990971 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.990811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-oauth-config\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.990971 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.990847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzcsr\" (UniqueName: \"kubernetes.io/projected/806c066d-d8dd-4b4b-aad5-23336f9e1248-kube-api-access-gzcsr\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.990971 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.990882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-oauth-serving-cert\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.990971 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.990924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-config\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.990971 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.990957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-serving-cert\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.991167 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.990985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-trusted-ca-bundle\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.991933 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.991911 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-trusted-ca-bundle\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.992306 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.992248 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-service-ca\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.992775 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.992753 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-oauth-serving-cert\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.993398 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.993353 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-config\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.994885 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.994845 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-oauth-config\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:53.995171 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:53.995145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-serving-cert\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:54.000955 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.000935 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzcsr\" (UniqueName: \"kubernetes.io/projected/806c066d-d8dd-4b4b-aad5-23336f9e1248-kube-api-access-gzcsr\") pod \"console-77ddd94b9d-68nkx\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:54.064027 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.063994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vsgtq" Jul 07 17:23:54.120391 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.120352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:23:54.134568 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.133602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkrbk" event={"ID":"0cba86e4-b5d0-446d-938b-ccf8dbc16c59","Type":"ContainerStarted","Data":"946f615cd566e125a2c3ca71a331a83d287a2ec147114fd2a234d625bf4c6353"} Jul 07 17:23:54.134568 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.133669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkrbk" event={"ID":"0cba86e4-b5d0-446d-938b-ccf8dbc16c59","Type":"ContainerStarted","Data":"2996b6fb497a4a440ffd4d60b0d69eacf89834d22654bb7794793a469247f6d8"} Jul 07 17:23:54.138957 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.138073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" event={"ID":"c023f6ac-d192-4c18-8d9e-04d029798295","Type":"ContainerStarted","Data":"4fd41228209e279502fdc21fbc887613f7568c5198da8b51ddd39e9f3fc17bea"} Jul 07 17:23:54.139948 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.139873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" event={"ID":"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf","Type":"ContainerStarted","Data":"42a5163c84c822a596cb0ca28d1f9bfec2a8d113794a16d1c5988620d1d4889b"} Jul 07 17:23:54.150785 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.148664 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:23:54.152080 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.152059 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.158195 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.156512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 17:23:54.158195 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.156730 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 17:23:54.158195 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.156919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 17:23:54.158195 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.157100 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 17:23:54.158195 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.157339 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 17:23:54.158195 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.157736 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 17:23:54.158195 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.158039 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 17:23:54.158532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.158234 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 17:23:54.158532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.158292 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6fsq6\"" Jul 07 17:23:54.158853 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.158639 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 17:23:54.158853 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.158661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 17:23:54.158853 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.158665 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2iqe9hnrasai\"" Jul 07 17:23:54.158853 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.158813 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 17:23:54.159137 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.159119 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 17:23:54.166515 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.166328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 17:23:54.182845 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.182475 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xkrbk" podStartSLOduration=66.640015909 podStartE2EDuration="1m8.182421709s" totalImagesPullingTime="1.5424058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:54.15791887 +0000 UTC m=+68.998232541" watchObservedRunningTime="2026-07-07 17:23:54.182421709 +0000 UTC m=+69.022735381" Jul 07 17:23:54.182967 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.182826 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 17:23:54.184447 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.184417 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 17:23:54.184649 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.184630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:23:54.192765 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.192857 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.192857 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.192975 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.192975 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.192975 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghxc\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-kube-api-access-jghxc\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.192975 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193118 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193118 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.192999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193118 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193118 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193118 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.193348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.193331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295089 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghxc\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-kube-api-access-jghxc\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295846 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295846 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295846 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.294894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295846 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.295161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295846 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.295627 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.295846 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.295828 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.297816 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.296289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.297816 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.297468 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.297816 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.297504 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.299351 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.299090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.303084 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.303060 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.305986 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.305966 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.307335 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.307293 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.307826 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.307787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.308091 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.308064 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.309049 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.309025 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.309147 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.309123 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.310358 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.310335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.311072 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.311030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.311677 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.311617 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.311677 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.311617 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.315668 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.315645 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.321871 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.321803 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghxc\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-kube-api-access-jghxc\") pod \"prometheus-k8s-0\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:54.469032 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:54.468259 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:23:57.810781 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:57.810744 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77ddd94b9d-68nkx"] Jul 07 17:23:58.132053 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:23:58.132004 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806c066d_d8dd_4b4b_aad5_23336f9e1248.slice/crio-6756d9fd87da5972b0d64ff46f38fe3ae83360b64a68537dfe7c0ba94e52d1de WatchSource:0}: Error finding container 6756d9fd87da5972b0d64ff46f38fe3ae83360b64a68537dfe7c0ba94e52d1de: Status 404 returned error can't find the container with id 6756d9fd87da5972b0d64ff46f38fe3ae83360b64a68537dfe7c0ba94e52d1de Jul 07 17:23:58.157368 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:58.157337 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddd94b9d-68nkx" event={"ID":"806c066d-d8dd-4b4b-aad5-23336f9e1248","Type":"ContainerStarted","Data":"6756d9fd87da5972b0d64ff46f38fe3ae83360b64a68537dfe7c0ba94e52d1de"} Jul 07 17:23:58.309203 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:58.309001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:23:59.161639 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.161607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29cf4" event={"ID":"de81cac2-c315-4825-927b-2b4c4f8f3790","Type":"ContainerStarted","Data":"57c2d8b19b51f7f43e206d61633acac269c8e1ef359bc7308efa91cb729674a9"} Jul 07 17:23:59.163419 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.163389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddd94b9d-68nkx" event={"ID":"806c066d-d8dd-4b4b-aad5-23336f9e1248","Type":"ContainerStarted","Data":"c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820"} Jul 07 17:23:59.164919 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.164897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f48zt" event={"ID":"5298398f-faa0-40fb-b6c8-420e6a8c7fb3","Type":"ContainerStarted","Data":"efd6673d92d0757ba1bc2e50890192329bf12a2bcec9d581ef72b09d97a55e12"} Jul 07 17:23:59.165025 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.164992 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:23:59.167303 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.167257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" event={"ID":"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade","Type":"ContainerStarted","Data":"2059dbb3bdc3483c0b6550179c3b4e1eb466b8fde1e199f2155ca5c601c3c729"} Jul 07 17:23:59.167411 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.167343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" event={"ID":"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade","Type":"ContainerStarted","Data":"2440fbe37dd6c8ede2a4a998f25bd3faa18b22f50ffca9a1cd5658b20cf48263"} Jul 07 17:23:59.167411 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.167384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" event={"ID":"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade","Type":"ContainerStarted","Data":"e2ef1e01069ae4e96122198bc63473172d171fdef41e6b054a6b6f611040d515"} Jul 07 17:23:59.169024 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.169005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" event={"ID":"c023f6ac-d192-4c18-8d9e-04d029798295","Type":"ContainerStarted","Data":"e5abf2f0f99ca3bd3889c7c98ff01bb4704cee3c29554332699c0ddf1fa3d45d"} Jul 07 17:23:59.169174 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.169138 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:59.171107 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.171084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" event={"ID":"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf","Type":"ContainerStarted","Data":"e7236662525b34c3956b2d0ff187925b7ec5c058d2a4261b9a942b1050cfcbf6"} Jul 07 17:23:59.172436 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.172415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerStarted","Data":"45f2fcdedf5e859e4a88373945a5b9a94bdd1713798641193671f02497debcc8"} Jul 07 17:23:59.175212 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.175192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" Jul 07 17:23:59.190388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.190060 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-29cf4" podStartSLOduration=66.990684632 podStartE2EDuration="1m13.190043688s" totalImagesPullingTime="6.199359056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:59.188743712 +0000 UTC m=+74.029057383" watchObservedRunningTime="2026-07-07 17:23:59.190043688 +0000 UTC m=+74.030357360" Jul 07 17:23:59.217501 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.217466 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-cxgql" podStartSLOduration=2.330803257 podStartE2EDuration="7.217452598s" totalImagesPullingTime="4.886649341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:59.217165578 +0000 UTC m=+74.057479249" watchObservedRunningTime="2026-07-07 17:23:59.217452598 +0000 UTC m=+74.057766285" Jul 07 17:23:59.258420 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:23:59.258381 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f48zt" podStartSLOduration=67.077191243 podStartE2EDuration="1m13.258365829s" totalImagesPullingTime="6.181174586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:59.256439266 +0000 UTC m=+74.096752937" watchObservedRunningTime="2026-07-07 17:23:59.258365829 +0000 UTC m=+74.098679532" Jul 07 17:24:00.181842 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.181752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" event={"ID":"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade","Type":"ContainerStarted","Data":"277baa1b0d6d4366d9482bc771d37a68850298936fe216c4e20e5c5f0ad2c48b"} Jul 07 17:24:00.181842 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.181825 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" event={"ID":"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade","Type":"ContainerStarted","Data":"9962b5345b1ab791bed2c578a8f15a7a73c32f41b3295ff2035308a0332a3cf8"} Jul 07 17:24:00.182308 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.181869 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:24:00.182308 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.181911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" event={"ID":"f4b1a2ef-8141-449c-99bf-0a2bd48b1ade","Type":"ContainerStarted","Data":"70774cfc3c458a9265476c5e976fba5094f333ed90f80f0451abca746babd743"} Jul 07 17:24:00.183992 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.183966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" event={"ID":"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf","Type":"ContainerStarted","Data":"c1b5b13281f5c29adcfbde460b07990211357565481b9414dea0b2bd7116ba5b"} Jul 07 17:24:00.184116 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.184029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" event={"ID":"4514d7be-35de-4eac-b3c1-0b9b3a41f9bf","Type":"ContainerStarted","Data":"d06c49d6126f4a342263e7a1c4b091dfaad690e20cce605aadc7ef5d2fa19455"} Jul 07 17:24:00.185543 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.185512 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerID="a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7" exitCode=0 Jul 07 17:24:00.185646 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.185608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7"} Jul 07 17:24:00.210466 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.210424 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77ddd94b9d-68nkx" podStartSLOduration=7.210411242 podStartE2EDuration="7.210411242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:23:59.284891829 +0000 UTC m=+74.125205500" watchObservedRunningTime="2026-07-07 17:24:00.210411242 +0000 UTC m=+75.050724910" Jul 07 17:24:00.258660 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.258622 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" podStartSLOduration=2.666760373 podStartE2EDuration="10.258607948s" totalImagesPullingTime="7.591847575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:24:00.215460264 +0000 UTC m=+75.055773933" watchObservedRunningTime="2026-07-07 17:24:00.258607948 +0000 UTC m=+75.098921619" Jul 07 17:24:00.259055 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:00.259035 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57fccdd65f-r8qmc" podStartSLOduration=1.180065854 podStartE2EDuration="7.259028928s" totalImagesPullingTime="6.078963074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:24:00.258286614 +0000 UTC m=+75.098600285" watchObservedRunningTime="2026-07-07 17:24:00.259028928 +0000 UTC m=+75.099342599" Jul 07 17:24:01.121082 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:01.121053 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77ddd94b9d-68nkx"] Jul 07 17:24:04.121508 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.121455 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:24:04.205505 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.205475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerStarted","Data":"cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d"} Jul 07 17:24:04.205639 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.205535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerStarted","Data":"cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b"} Jul 07 17:24:04.205639 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.205561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerStarted","Data":"c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574"} Jul 07 17:24:04.205639 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.205588 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerStarted","Data":"217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80"} Jul 07 17:24:04.205639 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.205610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerStarted","Data":"594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6"} Jul 07 17:24:04.205639 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.205635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerStarted","Data":"4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622"} Jul 07 17:24:04.262346 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.262285 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.276324993 podStartE2EDuration="10.262246867s" totalImagesPullingTime="4.968526701s" totalInitContainerRuntime="17.395173ms" isStatefulPod=false podCreationTimestamp="2026-07-07 17:23:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:24:04.261063315 +0000 UTC m=+79.101376986" watchObservedRunningTime="2026-07-07 17:24:04.262246867 +0000 UTC m=+79.102560541" Jul 07 17:24:04.469320 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:04.469284 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:24:06.195782 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:06.195651 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-789dd4f6c8-7lphr" Jul 07 17:24:11.763226 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:11.763188 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c59fbdcc4-bjs6m"] Jul 07 17:24:26.209817 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.209759 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-77ddd94b9d-68nkx" podUID="806c066d-d8dd-4b4b-aad5-23336f9e1248" containerName="console" containerID="cri-o://c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820" gracePeriod=15 Jul 07 17:24:26.444004 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.443982 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:24:26.557465 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-oauth-config\") pod \"806c066d-d8dd-4b4b-aad5-23336f9e1248\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " Jul 07 17:24:26.557583 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzcsr\" (UniqueName: \"kubernetes.io/projected/806c066d-d8dd-4b4b-aad5-23336f9e1248-kube-api-access-gzcsr\") pod \"806c066d-d8dd-4b4b-aad5-23336f9e1248\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " Jul 07 17:24:26.557583 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557507 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-oauth-serving-cert\") pod \"806c066d-d8dd-4b4b-aad5-23336f9e1248\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " Jul 07 17:24:26.557583 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557531 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-config\") pod \"806c066d-d8dd-4b4b-aad5-23336f9e1248\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " Jul 07 17:24:26.557583 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557579 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-trusted-ca-bundle\") pod \"806c066d-d8dd-4b4b-aad5-23336f9e1248\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " Jul 07 17:24:26.557788 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-serving-cert\") pod \"806c066d-d8dd-4b4b-aad5-23336f9e1248\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " Jul 07 17:24:26.557788 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-service-ca\") pod \"806c066d-d8dd-4b4b-aad5-23336f9e1248\" (UID: \"806c066d-d8dd-4b4b-aad5-23336f9e1248\") " Jul 07 17:24:26.557891 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557825 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "806c066d-d8dd-4b4b-aad5-23336f9e1248" (UID: "806c066d-d8dd-4b4b-aad5-23336f9e1248"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:26.558005 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.557975 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-config" (OuterVolumeSpecName: "console-config") pod "806c066d-d8dd-4b4b-aad5-23336f9e1248" (UID: "806c066d-d8dd-4b4b-aad5-23336f9e1248"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:26.558162 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.558021 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "806c066d-d8dd-4b4b-aad5-23336f9e1248" (UID: "806c066d-d8dd-4b4b-aad5-23336f9e1248"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:26.558400 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.558379 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-service-ca" (OuterVolumeSpecName: "service-ca") pod "806c066d-d8dd-4b4b-aad5-23336f9e1248" (UID: "806c066d-d8dd-4b4b-aad5-23336f9e1248"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:26.559973 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.559934 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806c066d-d8dd-4b4b-aad5-23336f9e1248-kube-api-access-gzcsr" (OuterVolumeSpecName: "kube-api-access-gzcsr") pod "806c066d-d8dd-4b4b-aad5-23336f9e1248" (UID: "806c066d-d8dd-4b4b-aad5-23336f9e1248"). InnerVolumeSpecName "kube-api-access-gzcsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:24:26.560062 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.559942 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "806c066d-d8dd-4b4b-aad5-23336f9e1248" (UID: "806c066d-d8dd-4b4b-aad5-23336f9e1248"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:26.560062 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.559988 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "806c066d-d8dd-4b4b-aad5-23336f9e1248" (UID: "806c066d-d8dd-4b4b-aad5-23336f9e1248"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:26.659112 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.659089 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-oauth-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:26.659112 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.659110 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzcsr\" (UniqueName: \"kubernetes.io/projected/806c066d-d8dd-4b4b-aad5-23336f9e1248-kube-api-access-gzcsr\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:26.659229 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.659119 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-oauth-serving-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:26.659229 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.659129 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:26.659229 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.659137 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-trusted-ca-bundle\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:26.659229 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.659145 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/806c066d-d8dd-4b4b-aad5-23336f9e1248-console-serving-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:26.659229 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:26.659155 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/806c066d-d8dd-4b4b-aad5-23336f9e1248-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:27.294563 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.294528 2579 generic.go:356] "Generic (PLEG): container finished" podID="806c066d-d8dd-4b4b-aad5-23336f9e1248" containerID="c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820" exitCode=0 Jul 07 17:24:27.294987 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.294569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddd94b9d-68nkx" event={"ID":"806c066d-d8dd-4b4b-aad5-23336f9e1248","Type":"ContainerDied","Data":"c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820"} Jul 07 17:24:27.294987 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.294576 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddd94b9d-68nkx" Jul 07 17:24:27.294987 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.294612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddd94b9d-68nkx" event={"ID":"806c066d-d8dd-4b4b-aad5-23336f9e1248","Type":"ContainerDied","Data":"6756d9fd87da5972b0d64ff46f38fe3ae83360b64a68537dfe7c0ba94e52d1de"} Jul 07 17:24:27.294987 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.294634 2579 scope.go:117] "RemoveContainer" containerID="c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820" Jul 07 17:24:27.304370 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.304354 2579 scope.go:117] "RemoveContainer" containerID="c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820" Jul 07 17:24:27.304664 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:24:27.304645 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820\": container with ID starting with c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820 not found: ID does not exist" containerID="c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820" Jul 07 17:24:27.304707 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.304675 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820"} err="failed to get container status \"c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820\": rpc error: code = NotFound desc = could not find container \"c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820\": container with ID starting with c72b4cc816778aca1bd2d6ce616145901b03a99fca36acd95209135aab940820 not found: ID does not exist" Jul 07 17:24:27.317168 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.317143 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77ddd94b9d-68nkx"] Jul 07 17:24:27.323211 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.323189 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77ddd94b9d-68nkx"] Jul 07 17:24:27.841809 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:27.841780 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="806c066d-d8dd-4b4b-aad5-23336f9e1248" path="/var/lib/kubelet/pods/806c066d-d8dd-4b4b-aad5-23336f9e1248/volumes" Jul 07 17:24:30.188889 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:30.188863 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f48zt" Jul 07 17:24:36.783835 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:36.783775 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c59fbdcc4-bjs6m" podUID="851db565-01a9-4ef5-be99-3853ce29fd2a" containerName="console" containerID="cri-o://dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13" gracePeriod=15 Jul 07 17:24:37.025190 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.025168 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:24:37.134366 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134247 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-oauth-serving-cert\") pod \"851db565-01a9-4ef5-be99-3853ce29fd2a\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " Jul 07 17:24:37.134366 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134330 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7h2z\" (UniqueName: \"kubernetes.io/projected/851db565-01a9-4ef5-be99-3853ce29fd2a-kube-api-access-b7h2z\") pod \"851db565-01a9-4ef5-be99-3853ce29fd2a\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " Jul 07 17:24:37.134366 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134361 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-serving-cert\") pod \"851db565-01a9-4ef5-be99-3853ce29fd2a\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " Jul 07 17:24:37.134632 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134391 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-console-config\") pod \"851db565-01a9-4ef5-be99-3853ce29fd2a\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " Jul 07 17:24:37.134632 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-oauth-config\") pod \"851db565-01a9-4ef5-be99-3853ce29fd2a\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " Jul 07 17:24:37.134632 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134472 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-service-ca\") pod \"851db565-01a9-4ef5-be99-3853ce29fd2a\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " Jul 07 17:24:37.134632 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134508 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-trusted-ca-bundle\") pod \"851db565-01a9-4ef5-be99-3853ce29fd2a\" (UID: \"851db565-01a9-4ef5-be99-3853ce29fd2a\") " Jul 07 17:24:37.134632 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134611 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "851db565-01a9-4ef5-be99-3853ce29fd2a" (UID: "851db565-01a9-4ef5-be99-3853ce29fd2a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:37.134863 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134760 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-oauth-serving-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:37.134922 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134879 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-console-config" (OuterVolumeSpecName: "console-config") pod "851db565-01a9-4ef5-be99-3853ce29fd2a" (UID: "851db565-01a9-4ef5-be99-3853ce29fd2a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:37.134922 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.134876 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-service-ca" (OuterVolumeSpecName: "service-ca") pod "851db565-01a9-4ef5-be99-3853ce29fd2a" (UID: "851db565-01a9-4ef5-be99-3853ce29fd2a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:37.135107 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.135088 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "851db565-01a9-4ef5-be99-3853ce29fd2a" (UID: "851db565-01a9-4ef5-be99-3853ce29fd2a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:24:37.137058 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.137022 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "851db565-01a9-4ef5-be99-3853ce29fd2a" (UID: "851db565-01a9-4ef5-be99-3853ce29fd2a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:37.137346 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.137330 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851db565-01a9-4ef5-be99-3853ce29fd2a-kube-api-access-b7h2z" (OuterVolumeSpecName: "kube-api-access-b7h2z") pod "851db565-01a9-4ef5-be99-3853ce29fd2a" (UID: "851db565-01a9-4ef5-be99-3853ce29fd2a"). InnerVolumeSpecName "kube-api-access-b7h2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:24:37.137405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.137351 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "851db565-01a9-4ef5-be99-3853ce29fd2a" (UID: "851db565-01a9-4ef5-be99-3853ce29fd2a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:24:37.235829 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.235795 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-oauth-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:37.235829 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.235823 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:37.235829 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.235835 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-trusted-ca-bundle\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:37.236060 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.235847 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7h2z\" (UniqueName: \"kubernetes.io/projected/851db565-01a9-4ef5-be99-3853ce29fd2a-kube-api-access-b7h2z\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:37.236060 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.235859 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/851db565-01a9-4ef5-be99-3853ce29fd2a-console-serving-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:37.236060 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.235870 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/851db565-01a9-4ef5-be99-3853ce29fd2a-console-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:24:37.335178 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.335141 2579 generic.go:356] "Generic (PLEG): container finished" podID="851db565-01a9-4ef5-be99-3853ce29fd2a" containerID="dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13" exitCode=0 Jul 07 17:24:37.335346 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.335191 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59fbdcc4-bjs6m" Jul 07 17:24:37.335346 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.335230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59fbdcc4-bjs6m" event={"ID":"851db565-01a9-4ef5-be99-3853ce29fd2a","Type":"ContainerDied","Data":"dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13"} Jul 07 17:24:37.335346 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.335301 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59fbdcc4-bjs6m" event={"ID":"851db565-01a9-4ef5-be99-3853ce29fd2a","Type":"ContainerDied","Data":"52e322ba2a8f9480a04abfec41910ed12b44f616bb758d4939a41b39b49a0c9a"} Jul 07 17:24:37.335346 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.335323 2579 scope.go:117] "RemoveContainer" containerID="dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13" Jul 07 17:24:37.343959 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.343943 2579 scope.go:117] "RemoveContainer" containerID="dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13" Jul 07 17:24:37.344212 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:24:37.344189 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13\": container with ID starting with dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13 not found: ID does not exist" containerID="dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13" Jul 07 17:24:37.344260 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.344223 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13"} err="failed to get container status \"dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13\": rpc error: code = NotFound desc = could not find container \"dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13\": container with ID starting with dbce89f9b1bcb3a3a1cb4ff8e1dad9e48fd8b1f65afa42e77c04d71a9aadcb13 not found: ID does not exist" Jul 07 17:24:37.356647 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.356625 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c59fbdcc4-bjs6m"] Jul 07 17:24:37.360780 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.360757 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c59fbdcc4-bjs6m"] Jul 07 17:24:37.841134 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:37.841099 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851db565-01a9-4ef5-be99-3853ce29fd2a" path="/var/lib/kubelet/pods/851db565-01a9-4ef5-be99-3853ce29fd2a/volumes" Jul 07 17:24:43.176539 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:43.176497 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rjwjb_4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f/kube-state-metrics/0.log" Jul 07 17:24:43.359726 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:43.359695 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rjwjb_4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f/kube-rbac-proxy-main/0.log" Jul 07 17:24:43.563120 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:43.563064 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rjwjb_4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f/kube-rbac-proxy-self/0.log" Jul 07 17:24:43.963448 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:43.963375 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-cxgql_c023f6ac-d192-4c18-8d9e-04d029798295/monitoring-plugin/0.log" Jul 07 17:24:45.362883 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:45.362859 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnjpm_7ba304d4-350f-4117-847c-c3ec3fd08f78/init-textfile/0.log" Jul 07 17:24:45.564638 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:45.564615 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnjpm_7ba304d4-350f-4117-847c-c3ec3fd08f78/node-exporter/0.log" Jul 07 17:24:45.761202 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:45.761183 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnjpm_7ba304d4-350f-4117-847c-c3ec3fd08f78/kube-rbac-proxy/0.log" Jul 07 17:24:45.961710 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:45.961679 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9nlxs_de194883-e38b-41b6-b9ec-aec7ffe6342b/kube-rbac-proxy-main/0.log" Jul 07 17:24:46.159691 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:46.159628 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9nlxs_de194883-e38b-41b6-b9ec-aec7ffe6342b/kube-rbac-proxy-self/0.log" Jul 07 17:24:46.361847 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:46.361823 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9nlxs_de194883-e38b-41b6-b9ec-aec7ffe6342b/openshift-state-metrics/0.log" Jul 07 17:24:46.567643 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:46.567623 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4154fe9-3a6a-4c46-bcdb-82fde031be82/init-config-reloader/0.log" Jul 07 17:24:46.763615 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:46.763595 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4154fe9-3a6a-4c46-bcdb-82fde031be82/prometheus/0.log" Jul 07 17:24:46.960371 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:46.960311 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4154fe9-3a6a-4c46-bcdb-82fde031be82/config-reloader/0.log" Jul 07 17:24:47.162530 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:47.162509 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4154fe9-3a6a-4c46-bcdb-82fde031be82/thanos-sidecar/0.log" Jul 07 17:24:47.360992 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:47.360966 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4154fe9-3a6a-4c46-bcdb-82fde031be82/kube-rbac-proxy-web/0.log" Jul 07 17:24:47.561516 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:47.561488 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4154fe9-3a6a-4c46-bcdb-82fde031be82/kube-rbac-proxy/0.log" Jul 07 17:24:47.759568 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:47.759539 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4154fe9-3a6a-4c46-bcdb-82fde031be82/kube-rbac-proxy-thanos/0.log" Jul 07 17:24:48.566615 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:48.566588 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57fccdd65f-r8qmc_4514d7be-35de-4eac-b3c1-0b9b3a41f9bf/telemeter-client/0.log" Jul 07 17:24:48.765784 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:48.765761 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57fccdd65f-r8qmc_4514d7be-35de-4eac-b3c1-0b9b3a41f9bf/reload/0.log" Jul 07 17:24:48.960688 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:48.960616 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57fccdd65f-r8qmc_4514d7be-35de-4eac-b3c1-0b9b3a41f9bf/kube-rbac-proxy/0.log" Jul 07 17:24:49.164975 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:49.164945 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/thanos-query/0.log" Jul 07 17:24:49.359985 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:49.359961 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy-web/0.log" Jul 07 17:24:49.561532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:49.561509 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy/0.log" Jul 07 17:24:49.760669 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:49.760640 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/prom-label-proxy/0.log" Jul 07 17:24:49.964313 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:49.964257 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy-rules/0.log" Jul 07 17:24:50.159588 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:50.159520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy-metrics/0.log" Jul 07 17:24:51.963178 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:51.963149 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z25lv_f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518/serve-healthcheck-canary/0.log" Jul 07 17:24:54.468423 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:54.468396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:24:54.490334 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:54.490312 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:24:55.429158 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:24:55.429134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:12.712081 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:12.712047 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:25:12.712662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:12.712338 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="prometheus" containerID="cri-o://4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622" gracePeriod=600 Jul 07 17:25:12.712662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:12.712412 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="thanos-sidecar" containerID="cri-o://217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80" gracePeriod=600 Jul 07 17:25:12.712662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:12.712404 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy" containerID="cri-o://cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b" gracePeriod=600 Jul 07 17:25:12.712662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:12.712426 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d" gracePeriod=600 Jul 07 17:25:12.712662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:12.712448 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-web" containerID="cri-o://c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574" gracePeriod=600 Jul 07 17:25:12.712662 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:12.712456 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="config-reloader" containerID="cri-o://594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6" gracePeriod=600 Jul 07 17:25:13.479307 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479258 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerID="cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d" exitCode=0 Jul 07 17:25:13.479307 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479298 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerID="cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b" exitCode=0 Jul 07 17:25:13.479307 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479305 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerID="217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80" exitCode=0 Jul 07 17:25:13.479307 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479298 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d"} Jul 07 17:25:13.479577 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b"} Jul 07 17:25:13.479577 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479311 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerID="594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6" exitCode=0 Jul 07 17:25:13.479577 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479391 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerID="4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622" exitCode=0 Jul 07 17:25:13.479577 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80"} Jul 07 17:25:13.479577 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6"} Jul 07 17:25:13.479577 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.479446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622"} Jul 07 17:25:13.956761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:13.956740 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.007406 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.007328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-db\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.007656 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.007604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.007963 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.007834 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-0\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.007963 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.007890 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-web-config\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008281 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config-out\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008281 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008241 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-metrics-client-certs\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008427 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008326 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jghxc\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-kube-api-access-jghxc\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008427 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008356 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-serving-certs-ca-bundle\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008427 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008422 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-kubelet-serving-ca-bundle\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008633 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008460 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-thanos-prometheus-http-client-file\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008633 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008492 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-tls-assets\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008633 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008527 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-grpc-tls\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008633 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008560 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-trusted-ca-bundle\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008633 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008589 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-kube-rbac-proxy\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008633 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008630 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-2\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008667 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008700 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-metrics-client-ca\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-tls\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.008917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.008831 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-1\") pod \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\" (UID: \"c4154fe9-3a6a-4c46-bcdb-82fde031be82\") " Jul 07 17:25:14.011129 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.009439 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:25:14.011229 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.011206 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:25:14.012444 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.012416 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:25:14.013210 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.012651 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:25:14.013453 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.013418 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:25:14.013874 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.013837 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:25:14.016198 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.016094 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:25:14.016910 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.016380 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.016910 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.016533 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:25:14.016910 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.016879 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:25:14.020285 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.018652 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.020285 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.018738 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config-out" (OuterVolumeSpecName: "config-out") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:25:14.020285 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.018909 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.020285 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.019040 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.020285 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.019879 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.020598 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.020370 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.020598 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.020452 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config" (OuterVolumeSpecName: "config") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.020711 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.020637 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.020844 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.020823 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-kube-api-access-jghxc" (OuterVolumeSpecName: "kube-api-access-jghxc") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "kube-api-access-jghxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:25:14.030699 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.030674 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-web-config" (OuterVolumeSpecName: "web-config") pod "c4154fe9-3a6a-4c46-bcdb-82fde031be82" (UID: "c4154fe9-3a6a-4c46-bcdb-82fde031be82"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110097 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110129 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110139 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110148 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-db\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110159 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110168 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110176 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-web-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110179 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110185 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config-out\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110192 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-metrics-client-certs\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110200 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jghxc\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-kube-api-access-jghxc\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110208 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110217 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110225 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110233 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4154fe9-3a6a-4c46-bcdb-82fde031be82-tls-assets\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110241 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-grpc-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110249 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110257 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-secret-kube-rbac-proxy\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110286 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110296 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4154fe9-3a6a-4c46-bcdb-82fde031be82-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.110532 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.110304 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4154fe9-3a6a-4c46-bcdb-82fde031be82-configmap-metrics-client-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:25:14.486400 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.486363 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerID="c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574" exitCode=0 Jul 07 17:25:14.486548 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.486416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574"} Jul 07 17:25:14.486548 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.486481 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.486548 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.486527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4154fe9-3a6a-4c46-bcdb-82fde031be82","Type":"ContainerDied","Data":"45f2fcdedf5e859e4a88373945a5b9a94bdd1713798641193671f02497debcc8"} Jul 07 17:25:14.486653 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.486579 2579 scope.go:117] "RemoveContainer" containerID="cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d" Jul 07 17:25:14.495383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.495359 2579 scope.go:117] "RemoveContainer" containerID="cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b" Jul 07 17:25:14.502944 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.502925 2579 scope.go:117] "RemoveContainer" containerID="c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574" Jul 07 17:25:14.511389 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.511372 2579 scope.go:117] "RemoveContainer" containerID="217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80" Jul 07 17:25:14.514261 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.514239 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:25:14.518534 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.518510 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:25:14.519617 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.519600 2579 scope.go:117] "RemoveContainer" containerID="594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6" Jul 07 17:25:14.526101 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.526082 2579 scope.go:117] "RemoveContainer" containerID="4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622" Jul 07 17:25:14.533015 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.532998 2579 scope.go:117] "RemoveContainer" containerID="a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7" Jul 07 17:25:14.539255 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.539239 2579 scope.go:117] "RemoveContainer" containerID="cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d" Jul 07 17:25:14.539504 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:25:14.539486 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d\": container with ID starting with cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d not found: ID does not exist" containerID="cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d" Jul 07 17:25:14.539549 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.539511 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d"} err="failed to get container status \"cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d\": rpc error: code = NotFound desc = could not find container \"cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d\": container with ID starting with cb5b2b9919aff319cb9c82383c8823029a6b5074ee622323084fa6e0a9a7b28d not found: ID does not exist" Jul 07 17:25:14.539549 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.539528 2579 scope.go:117] "RemoveContainer" containerID="cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b" Jul 07 17:25:14.539745 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:25:14.539726 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b\": container with ID starting with cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b not found: ID does not exist" containerID="cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b" Jul 07 17:25:14.539780 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.539756 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b"} err="failed to get container status \"cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b\": rpc error: code = NotFound desc = could not find container \"cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b\": container with ID starting with cfc88d4887117090359f20d915dd0b7c5076183622be4f90e1309873f428a27b not found: ID does not exist" Jul 07 17:25:14.539780 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.539775 2579 scope.go:117] "RemoveContainer" containerID="c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574" Jul 07 17:25:14.540001 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:25:14.539985 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574\": container with ID starting with c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574 not found: ID does not exist" containerID="c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574" Jul 07 17:25:14.540057 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540004 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574"} err="failed to get container status \"c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574\": rpc error: code = NotFound desc = could not find container \"c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574\": container with ID starting with c55bc7f338e75fe8eb5898f31e0a826003615b7192967b8d76830211e69c6574 not found: ID does not exist" Jul 07 17:25:14.540057 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540017 2579 scope.go:117] "RemoveContainer" containerID="217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80" Jul 07 17:25:14.540197 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:25:14.540178 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80\": container with ID starting with 217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80 not found: ID does not exist" containerID="217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80" Jul 07 17:25:14.540235 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540201 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80"} err="failed to get container status \"217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80\": rpc error: code = NotFound desc = could not find container \"217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80\": container with ID starting with 217a132edfb620c09a9ab893040583f69744bebc395d452afc07ed8a1bbb2e80 not found: ID does not exist" Jul 07 17:25:14.540235 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540213 2579 scope.go:117] "RemoveContainer" containerID="594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6" Jul 07 17:25:14.540695 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:25:14.540678 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6\": container with ID starting with 594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6 not found: ID does not exist" containerID="594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6" Jul 07 17:25:14.540766 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540702 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6"} err="failed to get container status \"594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6\": rpc error: code = NotFound desc = could not find container \"594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6\": container with ID starting with 594163e522af0716c25c141862aab15c30f242d3770ac1dc6e118d3920040ea6 not found: ID does not exist" Jul 07 17:25:14.540766 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540724 2579 scope.go:117] "RemoveContainer" containerID="4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622" Jul 07 17:25:14.540944 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:25:14.540929 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622\": container with ID starting with 4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622 not found: ID does not exist" containerID="4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622" Jul 07 17:25:14.540982 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540948 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622"} err="failed to get container status \"4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622\": rpc error: code = NotFound desc = could not find container \"4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622\": container with ID starting with 4fff70fe8a058f56152e5fe566b0623026c065e8026427b26524c3b78484a622 not found: ID does not exist" Jul 07 17:25:14.540982 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.540967 2579 scope.go:117] "RemoveContainer" containerID="a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7" Jul 07 17:25:14.541221 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:25:14.541205 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7\": container with ID starting with a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7 not found: ID does not exist" containerID="a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7" Jul 07 17:25:14.541305 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.541225 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7"} err="failed to get container status \"a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7\": rpc error: code = NotFound desc = could not find container \"a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7\": container with ID starting with a4051de2ca486d175a14fe37298bc5bd20664c1f7aa95802559fbd81dbd4aeb7 not found: ID does not exist" Jul 07 17:25:14.555175 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555156 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:25:14.555426 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555414 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="prometheus" Jul 07 17:25:14.555465 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555428 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="prometheus" Jul 07 17:25:14.555465 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555441 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851db565-01a9-4ef5-be99-3853ce29fd2a" containerName="console" Jul 07 17:25:14.555465 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555447 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="851db565-01a9-4ef5-be99-3853ce29fd2a" containerName="console" Jul 07 17:25:14.555465 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555454 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="config-reloader" Jul 07 17:25:14.555465 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555459 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="config-reloader" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555470 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555475 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555483 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="init-config-reloader" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555488 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="init-config-reloader" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555493 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-thanos" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555498 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-thanos" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555504 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806c066d-d8dd-4b4b-aad5-23336f9e1248" containerName="console" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555509 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="806c066d-d8dd-4b4b-aad5-23336f9e1248" containerName="console" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555513 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="thanos-sidecar" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555518 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="thanos-sidecar" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555523 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-web" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555528 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-web" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555570 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="thanos-sidecar" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555576 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="prometheus" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555583 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555588 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-thanos" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555594 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="806c066d-d8dd-4b4b-aad5-23336f9e1248" containerName="console" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555599 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="851db565-01a9-4ef5-be99-3853ce29fd2a" containerName="console" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555605 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="config-reloader" Jul 07 17:25:14.555610 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.555610 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" containerName="kube-rbac-proxy-web" Jul 07 17:25:14.559098 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.559083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.563202 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.563188 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 17:25:14.565460 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565437 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 17:25:14.565565 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565506 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 17:25:14.565629 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565609 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 17:25:14.565754 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565735 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 17:25:14.565837 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565783 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 17:25:14.565837 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565736 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 17:25:14.565935 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565848 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 17:25:14.565935 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565908 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 17:25:14.565935 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.565929 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 17:25:14.566481 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.566466 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6fsq6\"" Jul 07 17:25:14.566678 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.566654 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 17:25:14.566745 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.566696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2iqe9hnrasai\"" Jul 07 17:25:14.566792 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.566697 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 17:25:14.566827 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.566817 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 17:25:14.568524 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.568506 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 17:25:14.572668 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.572648 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 17:25:14.580433 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.580414 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:25:14.612983 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.612962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7fd3faf-15bd-4af7-bb5b-617b476c782a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.612983 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.612985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7fd3faf-15bd-4af7-bb5b-617b476c782a-config-out\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613114 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613114 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613114 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613114 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613316 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613316 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613316 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613316 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613316 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613316 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tst\" (UniqueName: \"kubernetes.io/projected/d7fd3faf-15bd-4af7-bb5b-617b476c782a-kube-api-access-z4tst\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-config\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-web-config\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.613525 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.613494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714371 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714371 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714515 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tst\" (UniqueName: \"kubernetes.io/projected/d7fd3faf-15bd-4af7-bb5b-617b476c782a-kube-api-access-z4tst\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714515 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714622 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714622 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714723 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714723 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-config\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714723 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714723 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-web-config\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714729 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7fd3faf-15bd-4af7-bb5b-617b476c782a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7fd3faf-15bd-4af7-bb5b-617b476c782a-config-out\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.714917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.715323 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.715323 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.714973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.715323 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.715001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.715323 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.715037 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.715998 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.715473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.715998 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.715651 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.715998 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.715794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.716146 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.716002 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.716917 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.716891 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.717612 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.717594 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.718385 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.717766 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.718560 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.718305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7fd3faf-15bd-4af7-bb5b-617b476c782a-config-out\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.718935 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.718908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7fd3faf-15bd-4af7-bb5b-617b476c782a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.719028 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.718984 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7fd3faf-15bd-4af7-bb5b-617b476c782a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.719492 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.719468 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-config\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.719986 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.719959 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.720051 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.720020 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.720247 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.720223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.720800 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.720777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.720869 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.720812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-web-config\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.721156 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.721136 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7fd3faf-15bd-4af7-bb5b-617b476c782a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.725137 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.725120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tst\" (UniqueName: \"kubernetes.io/projected/d7fd3faf-15bd-4af7-bb5b-617b476c782a-kube-api-access-z4tst\") pod \"prometheus-k8s-0\" (UID: \"d7fd3faf-15bd-4af7-bb5b-617b476c782a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:14.868782 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:14.868750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:25:15.013626 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:15.013590 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:25:15.017380 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:25:15.017348 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fd3faf_15bd_4af7_bb5b_617b476c782a.slice/crio-9bbe4602e9beee35cdb9e490ceeab691163b2a755752e1c4f828cc54c5c9e074 WatchSource:0}: Error finding container 9bbe4602e9beee35cdb9e490ceeab691163b2a755752e1c4f828cc54c5c9e074: Status 404 returned error can't find the container with id 9bbe4602e9beee35cdb9e490ceeab691163b2a755752e1c4f828cc54c5c9e074 Jul 07 17:25:15.492282 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:15.492240 2579 generic.go:356] "Generic (PLEG): container finished" podID="d7fd3faf-15bd-4af7-bb5b-617b476c782a" containerID="67fcc97caefe4804c6343d942938395ba24878e203bcd6f02ff5010a8af6b4a1" exitCode=0 Jul 07 17:25:15.492427 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:15.492314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerDied","Data":"67fcc97caefe4804c6343d942938395ba24878e203bcd6f02ff5010a8af6b4a1"} Jul 07 17:25:15.492427 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:15.492371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerStarted","Data":"9bbe4602e9beee35cdb9e490ceeab691163b2a755752e1c4f828cc54c5c9e074"} Jul 07 17:25:15.844155 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:15.844123 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4154fe9-3a6a-4c46-bcdb-82fde031be82" path="/var/lib/kubelet/pods/c4154fe9-3a6a-4c46-bcdb-82fde031be82/volumes" Jul 07 17:25:16.499501 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:16.499467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerStarted","Data":"993c94fccaec1d7d1a86c9c8f1d62087a4f3caf76ec754a47a080ab7814e650a"} Jul 07 17:25:16.499860 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:16.499530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerStarted","Data":"06f826139374c635921e21a3693f4cc7f41117478fb6c3068e74a5866f25e48f"} Jul 07 17:25:16.499860 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:16.499556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerStarted","Data":"55f0dd9ee09f24acbc3e6647f770110ee6f86bf0ac3e3d3568b78c3717a9489e"} Jul 07 17:25:16.499860 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:16.499579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerStarted","Data":"aae91a4ca049e5356ecb204ab3db857422be1ac76be9e9dd1c729091b542af54"} Jul 07 17:25:16.499860 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:16.499606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerStarted","Data":"df5186146535e3e9fa4d0c1fd69067ec24a8757d5dee1653db0ddf12cbfc1179"} Jul 07 17:25:16.499860 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:16.499631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7fd3faf-15bd-4af7-bb5b-617b476c782a","Type":"ContainerStarted","Data":"30243a99cbece790b67c9f57851849b5f4b3b61c5a9d16b35596e7d5b3a5ac59"} Jul 07 17:25:16.547912 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:16.547858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.529052493 podStartE2EDuration="2.547842141s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.789648ms" isStatefulPod=false podCreationTimestamp="2026-07-07 17:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:25:16.546496236 +0000 UTC m=+151.386809908" watchObservedRunningTime="2026-07-07 17:25:16.547842141 +0000 UTC m=+151.388155812" Jul 07 17:25:19.869131 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:25:19.869089 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:26:14.869344 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:26:14.869296 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:26:14.885244 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:26:14.885221 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:26:15.743420 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:26:15.743392 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:27:45.562713 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:27:45.562690 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 17:31:15.588076 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.588044 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4"] Jul 07 17:31:15.590457 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.590442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:15.593542 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.593514 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wcqlz\"" Jul 07 17:31:15.594422 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.594404 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 17:31:15.594544 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.594423 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:31:15.605777 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.605754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4"] Jul 07 17:31:15.666383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.666355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrst\" (UniqueName: \"kubernetes.io/projected/24b2c9d3-aaa3-4037-bf4c-2f62f08e5048-kube-api-access-frrst\") pod \"openshift-lws-operator-fd8ccff4c-gzpd4\" (UID: \"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:15.666506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.666434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b2c9d3-aaa3-4037-bf4c-2f62f08e5048-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gzpd4\" (UID: \"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:15.766745 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.766720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrst\" (UniqueName: \"kubernetes.io/projected/24b2c9d3-aaa3-4037-bf4c-2f62f08e5048-kube-api-access-frrst\") pod \"openshift-lws-operator-fd8ccff4c-gzpd4\" (UID: \"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:15.766886 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.766774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b2c9d3-aaa3-4037-bf4c-2f62f08e5048-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gzpd4\" (UID: \"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:15.767116 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.767087 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b2c9d3-aaa3-4037-bf4c-2f62f08e5048-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gzpd4\" (UID: \"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:15.778666 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.778646 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrst\" (UniqueName: \"kubernetes.io/projected/24b2c9d3-aaa3-4037-bf4c-2f62f08e5048-kube-api-access-frrst\") pod \"openshift-lws-operator-fd8ccff4c-gzpd4\" (UID: \"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:15.908835 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:15.908750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" Jul 07 17:31:16.037715 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:16.037646 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4"] Jul 07 17:31:16.040885 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:31:16.040842 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24b2c9d3_aaa3_4037_bf4c_2f62f08e5048.slice/crio-441fe4e2f76319793535dd3769b6327634598644057d93040cc614521f4d1b0f WatchSource:0}: Error finding container 441fe4e2f76319793535dd3769b6327634598644057d93040cc614521f4d1b0f: Status 404 returned error can't find the container with id 441fe4e2f76319793535dd3769b6327634598644057d93040cc614521f4d1b0f Jul 07 17:31:16.043292 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:16.043256 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:31:16.841653 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:16.841620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" event={"ID":"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048","Type":"ContainerStarted","Data":"441fe4e2f76319793535dd3769b6327634598644057d93040cc614521f4d1b0f"} Jul 07 17:31:19.860260 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:19.860227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" event={"ID":"24b2c9d3-aaa3-4037-bf4c-2f62f08e5048","Type":"ContainerStarted","Data":"1e9d2210f5c79341a4e4bb76b587aadaea39b24fcfae14fb91286e232ded7d47"} Jul 07 17:31:19.881007 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:19.880964 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzpd4" podStartSLOduration=1.862681821 podStartE2EDuration="4.8809493s" totalImagesPullingTime="3.018267479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:31:19.879225073 +0000 UTC m=+514.719538744" watchObservedRunningTime="2026-07-07 17:31:19.8809493 +0000 UTC m=+514.721262971" Jul 07 17:31:38.482635 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.482593 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd"] Jul 07 17:31:38.486329 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.486307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.494405 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.494387 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 17:31:38.495487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.495466 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 17:31:38.495586 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.495560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 17:31:38.498199 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.498181 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 17:31:38.516471 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.516455 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cpf46\"" Jul 07 17:31:38.524417 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.524396 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd"] Jul 07 17:31:38.638046 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.638020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b6e072c-8116-4676-a8fd-caf8e707f408-webhook-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.638139 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.638054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b6e072c-8116-4676-a8fd-caf8e707f408-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.638187 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.638136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6e072c-8116-4676-a8fd-caf8e707f408-tmp\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.638187 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.638154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b6e072c-8116-4676-a8fd-caf8e707f408-manifests\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.638187 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.638169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wnc\" (UniqueName: \"kubernetes.io/projected/3b6e072c-8116-4676-a8fd-caf8e707f408-kube-api-access-l7wnc\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.739349 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.739326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b6e072c-8116-4676-a8fd-caf8e707f408-webhook-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.739433 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.739357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b6e072c-8116-4676-a8fd-caf8e707f408-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.739433 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.739410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6e072c-8116-4676-a8fd-caf8e707f408-tmp\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.739433 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.739430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b6e072c-8116-4676-a8fd-caf8e707f408-manifests\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.739571 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.739446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wnc\" (UniqueName: \"kubernetes.io/projected/3b6e072c-8116-4676-a8fd-caf8e707f408-kube-api-access-l7wnc\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.739872 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.739853 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b6e072c-8116-4676-a8fd-caf8e707f408-manifests\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.741859 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.741826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6e072c-8116-4676-a8fd-caf8e707f408-tmp\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.741969 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.741913 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b6e072c-8116-4676-a8fd-caf8e707f408-webhook-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.742031 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.742007 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b6e072c-8116-4676-a8fd-caf8e707f408-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.751343 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.751323 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wnc\" (UniqueName: \"kubernetes.io/projected/3b6e072c-8116-4676-a8fd-caf8e707f408-kube-api-access-l7wnc\") pod \"opendatahub-operator-controller-manager-684cdbc7fc-2ddkd\" (UID: \"3b6e072c-8116-4676-a8fd-caf8e707f408\") " pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.797319 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.797299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:38.968761 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:38.968624 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd"] Jul 07 17:31:38.971404 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:31:38.971379 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6e072c_8116_4676_a8fd_caf8e707f408.slice/crio-84e6462ac6c237242f01709f6731792d295c2af171cdde92c277807550c7496e WatchSource:0}: Error finding container 84e6462ac6c237242f01709f6731792d295c2af171cdde92c277807550c7496e: Status 404 returned error can't find the container with id 84e6462ac6c237242f01709f6731792d295c2af171cdde92c277807550c7496e Jul 07 17:31:39.937015 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:39.936975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" event={"ID":"3b6e072c-8116-4676-a8fd-caf8e707f408","Type":"ContainerStarted","Data":"84e6462ac6c237242f01709f6731792d295c2af171cdde92c277807550c7496e"} Jul 07 17:31:42.950243 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:42.950205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" event={"ID":"3b6e072c-8116-4676-a8fd-caf8e707f408","Type":"ContainerStarted","Data":"072e3687fc47b4ea3b0b779c89609f12bec49ad17395c269c1b7954f2221dc43"} Jul 07 17:31:43.955994 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:43.955961 2579 generic.go:356] "Generic (PLEG): container finished" podID="3b6e072c-8116-4676-a8fd-caf8e707f408" containerID="072e3687fc47b4ea3b0b779c89609f12bec49ad17395c269c1b7954f2221dc43" exitCode=0 Jul 07 17:31:43.956402 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:43.956044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" event={"ID":"3b6e072c-8116-4676-a8fd-caf8e707f408","Type":"ContainerDied","Data":"072e3687fc47b4ea3b0b779c89609f12bec49ad17395c269c1b7954f2221dc43"} Jul 07 17:31:46.969696 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:46.969652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" event={"ID":"3b6e072c-8116-4676-a8fd-caf8e707f408","Type":"ContainerStarted","Data":"00f5b340449ea0ed93d8108e75d6e3ff6b363b39bdb4a6b256680d2e1d4d20f6"} Jul 07 17:31:46.970094 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:46.969883 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:31:47.084373 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:47.084326 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" podStartSLOduration=2.691557592 podStartE2EDuration="9.084310571s" totalImagesPullingTime="5.302010017s" totalInitContainerRuntime="1.090742962s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:31:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:31:47.082585098 +0000 UTC m=+541.922898769" watchObservedRunningTime="2026-07-07 17:31:47.084310571 +0000 UTC m=+541.924624242" Jul 07 17:31:57.975282 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:31:57.975240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-684cdbc7fc-2ddkd" Jul 07 17:32:00.553888 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.553844 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-668c694757-ftrl5"] Jul 07 17:32:00.561260 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.561239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.588631 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.588607 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 17:32:00.589805 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.589789 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 17:32:00.591043 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.591025 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 17:32:00.591673 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.591659 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 17:32:00.591673 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.591664 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-glq4f\"" Jul 07 17:32:00.598740 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.598724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d9be73-4207-4fa0-99a9-5dba5872ae20-tls-certs\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.598817 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.598747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjrx\" (UniqueName: \"kubernetes.io/projected/29d9be73-4207-4fa0-99a9-5dba5872ae20-kube-api-access-jfjrx\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.598817 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.598797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d9be73-4207-4fa0-99a9-5dba5872ae20-tmp\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.631102 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.631074 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-668c694757-ftrl5"] Jul 07 17:32:00.699057 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.699031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d9be73-4207-4fa0-99a9-5dba5872ae20-tls-certs\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.699159 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.699062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjrx\" (UniqueName: \"kubernetes.io/projected/29d9be73-4207-4fa0-99a9-5dba5872ae20-kube-api-access-jfjrx\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.699159 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.699142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d9be73-4207-4fa0-99a9-5dba5872ae20-tmp\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.701421 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.701399 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d9be73-4207-4fa0-99a9-5dba5872ae20-tmp\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.701755 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.701736 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d9be73-4207-4fa0-99a9-5dba5872ae20-tls-certs\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.742972 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.742951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjrx\" (UniqueName: \"kubernetes.io/projected/29d9be73-4207-4fa0-99a9-5dba5872ae20-kube-api-access-jfjrx\") pod \"kube-auth-proxy-668c694757-ftrl5\" (UID: \"29d9be73-4207-4fa0-99a9-5dba5872ae20\") " pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:00.870322 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:00.870245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" Jul 07 17:32:01.031881 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:01.031852 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-668c694757-ftrl5"] Jul 07 17:32:01.044175 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:32:01.044149 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d9be73_4207_4fa0_99a9_5dba5872ae20.slice/crio-1ec2512dc9052588aacd0a2eea6022ef2074b1863576821db231e53306c5e502 WatchSource:0}: Error finding container 1ec2512dc9052588aacd0a2eea6022ef2074b1863576821db231e53306c5e502: Status 404 returned error can't find the container with id 1ec2512dc9052588aacd0a2eea6022ef2074b1863576821db231e53306c5e502 Jul 07 17:32:02.026048 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:02.026008 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" event={"ID":"29d9be73-4207-4fa0-99a9-5dba5872ae20","Type":"ContainerStarted","Data":"1ec2512dc9052588aacd0a2eea6022ef2074b1863576821db231e53306c5e502"} Jul 07 17:32:05.045614 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:05.045583 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" event={"ID":"29d9be73-4207-4fa0-99a9-5dba5872ae20","Type":"ContainerStarted","Data":"dab10379e10e734e4942f772999582a52fd2cf8e94dee24f66404426f37228aa"} Jul 07 17:32:11.200094 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.200037 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-668c694757-ftrl5" podStartSLOduration=8.067561699 podStartE2EDuration="11.200020914s" totalImagesPullingTime="3.132459215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:32:05.073201376 +0000 UTC m=+559.913515047" watchObservedRunningTime="2026-07-07 17:32:11.200020914 +0000 UTC m=+566.040334585" Jul 07 17:32:11.201211 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.201191 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s"] Jul 07 17:32:11.203422 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.203406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.209237 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.209206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 17:32:11.209399 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.209209 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 17:32:11.210139 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.210122 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5lghc\"" Jul 07 17:32:11.211158 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.211137 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 17:32:11.211382 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.211368 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 17:32:11.237798 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.237773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s"] Jul 07 17:32:11.285763 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.285733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6332754-e093-42fe-95dd-36741751618a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.285763 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.285764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.285951 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.285785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzhb\" (UniqueName: \"kubernetes.io/projected/f6332754-e093-42fe-95dd-36741751618a-kube-api-access-gwzhb\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.285951 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.285836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6332754-e093-42fe-95dd-36741751618a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.285951 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.285886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.285951 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.285922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6332754-e093-42fe-95dd-36741751618a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.286088 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.286004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.386832 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.386804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6332754-e093-42fe-95dd-36741751618a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.386948 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.386834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.386948 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.386860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzhb\" (UniqueName: \"kubernetes.io/projected/f6332754-e093-42fe-95dd-36741751618a-kube-api-access-gwzhb\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.387065 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.387048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6332754-e093-42fe-95dd-36741751618a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.387120 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.387104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.387177 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.387164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6332754-e093-42fe-95dd-36741751618a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.387288 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.387251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.387802 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.387711 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6332754-e093-42fe-95dd-36741751618a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.389417 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.389389 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.389514 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.389420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6332754-e093-42fe-95dd-36741751618a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.389618 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.389600 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.389618 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.389607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6332754-e093-42fe-95dd-36741751618a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.414186 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.414163 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzhb\" (UniqueName: \"kubernetes.io/projected/f6332754-e093-42fe-95dd-36741751618a-kube-api-access-gwzhb\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.417846 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.417825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6332754-e093-42fe-95dd-36741751618a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6zh2s\" (UID: \"f6332754-e093-42fe-95dd-36741751618a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.512261 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.512233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:11.685792 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:11.685768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s"] Jul 07 17:32:11.688444 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:32:11.688418 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6332754_e093_42fe_95dd_36741751618a.slice/crio-c568481a2e9d66b26e65ba651d5979d9b56882f93d6ca8ba4a63be0151861b14 WatchSource:0}: Error finding container c568481a2e9d66b26e65ba651d5979d9b56882f93d6ca8ba4a63be0151861b14: Status 404 returned error can't find the container with id c568481a2e9d66b26e65ba651d5979d9b56882f93d6ca8ba4a63be0151861b14 Jul 07 17:32:12.076712 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:12.076675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" event={"ID":"f6332754-e093-42fe-95dd-36741751618a","Type":"ContainerStarted","Data":"c568481a2e9d66b26e65ba651d5979d9b56882f93d6ca8ba4a63be0151861b14"} Jul 07 17:32:13.924584 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:13.924552 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw"] Jul 07 17:32:13.936505 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:13.936483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:13.947548 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:13.947522 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 17:32:13.947818 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:13.947800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 17:32:13.948643 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:13.948622 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gc8bv\"" Jul 07 17:32:13.948878 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:13.948845 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 17:32:13.950298 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:13.950250 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw"] Jul 07 17:32:14.012053 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.012019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8eb9601b-1989-4605-b25e-c794bf0723b2-metrics-cert\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.012236 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.012085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8eb9601b-1989-4605-b25e-c794bf0723b2-manager-config\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.012236 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.012134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbq6\" (UniqueName: \"kubernetes.io/projected/8eb9601b-1989-4605-b25e-c794bf0723b2-kube-api-access-hhbq6\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.012236 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.012183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eb9601b-1989-4605-b25e-c794bf0723b2-cert\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.113086 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.113056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbq6\" (UniqueName: \"kubernetes.io/projected/8eb9601b-1989-4605-b25e-c794bf0723b2-kube-api-access-hhbq6\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.113239 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.113102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eb9601b-1989-4605-b25e-c794bf0723b2-cert\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.113239 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.113162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8eb9601b-1989-4605-b25e-c794bf0723b2-metrics-cert\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.113239 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.113189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8eb9601b-1989-4605-b25e-c794bf0723b2-manager-config\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.113810 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.113787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8eb9601b-1989-4605-b25e-c794bf0723b2-manager-config\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.115548 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.115528 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8eb9601b-1989-4605-b25e-c794bf0723b2-metrics-cert\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.115639 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.115590 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eb9601b-1989-4605-b25e-c794bf0723b2-cert\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.144997 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.144964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbq6\" (UniqueName: \"kubernetes.io/projected/8eb9601b-1989-4605-b25e-c794bf0723b2-kube-api-access-hhbq6\") pod \"lws-controller-manager-686dcd8c67-qvhsw\" (UID: \"8eb9601b-1989-4605-b25e-c794bf0723b2\") " pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.250498 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.250475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:14.436522 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.436501 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw"] Jul 07 17:32:14.604442 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:32:14.604384 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb9601b_1989_4605_b25e_c794bf0723b2.slice/crio-55ac8d5a5378790902b4d6554387a5305f7ad71dac965770aaf05603bc755211 WatchSource:0}: Error finding container 55ac8d5a5378790902b4d6554387a5305f7ad71dac965770aaf05603bc755211: Status 404 returned error can't find the container with id 55ac8d5a5378790902b4d6554387a5305f7ad71dac965770aaf05603bc755211 Jul 07 17:32:14.640980 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.640944 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 07 17:32:14.641072 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:14.641010 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 07 17:32:15.092349 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:15.092303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" event={"ID":"8eb9601b-1989-4605-b25e-c794bf0723b2","Type":"ContainerStarted","Data":"55ac8d5a5378790902b4d6554387a5305f7ad71dac965770aaf05603bc755211"} Jul 07 17:32:15.094035 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:15.093997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" event={"ID":"f6332754-e093-42fe-95dd-36741751618a","Type":"ContainerStarted","Data":"a910952da8c3615c35fecd2aa748489dca522b81ad58822ceded78ffaa94aa82"} Jul 07 17:32:15.094214 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:15.094188 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:15.096076 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:15.096043 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-6zh2s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 17:32:15.096210 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:15.096107 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" podUID="f6332754-e093-42fe-95dd-36741751618a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 17:32:15.144253 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:15.144203 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" podStartSLOduration=1.193977154 podStartE2EDuration="4.144182905s" totalImagesPullingTime="2.950205751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:32:15.143702502 +0000 UTC m=+569.984016173" watchObservedRunningTime="2026-07-07 17:32:15.144182905 +0000 UTC m=+569.984496577" Jul 07 17:32:16.099043 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:16.099012 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6zh2s" Jul 07 17:32:18.108705 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:18.108677 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" event={"ID":"8eb9601b-1989-4605-b25e-c794bf0723b2","Type":"ContainerStarted","Data":"321ff6f39a0fb3fc8783311f95b623dbcd0854891ad0fe2927396105f4517e9c"} Jul 07 17:32:18.109117 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:18.108803 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:32:18.152254 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:18.152209 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" podStartSLOduration=2.37012092 podStartE2EDuration="5.152191684s" totalImagesPullingTime="2.782070764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:32:18.151679719 +0000 UTC m=+572.991993390" watchObservedRunningTime="2026-07-07 17:32:18.152191684 +0000 UTC m=+572.992505356" Jul 07 17:32:29.114723 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:32:29.114688 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-686dcd8c67-qvhsw" Jul 07 17:37:28.093482 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.093453 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v"] Jul 07 17:37:28.096855 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.096839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.100623 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.100598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 17:37:28.100623 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.100619 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 17:37:28.100868 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.100653 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 17:37:28.100868 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.100673 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bl7jx\"" Jul 07 17:37:28.107481 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.107460 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v"] Jul 07 17:37:28.214495 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.214468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.214495 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.214494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd684d89-b605-4a33-b94a-2f1002662d2a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.214673 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.214515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.214673 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.214604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqskx\" (UniqueName: \"kubernetes.io/projected/fd684d89-b605-4a33-b94a-2f1002662d2a-kube-api-access-wqskx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.214673 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.214642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.214673 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.214659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315238 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd684d89-b605-4a33-b94a-2f1002662d2a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqskx\" (UniqueName: \"kubernetes.io/projected/fd684d89-b605-4a33-b94a-2f1002662d2a-kube-api-access-wqskx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315800 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.315859 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315804 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.316030 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.315970 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.318182 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.318165 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd684d89-b605-4a33-b94a-2f1002662d2a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.318586 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.318565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd684d89-b605-4a33-b94a-2f1002662d2a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.325658 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.325634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqskx\" (UniqueName: \"kubernetes.io/projected/fd684d89-b605-4a33-b94a-2f1002662d2a-kube-api-access-wqskx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v\" (UID: \"fd684d89-b605-4a33-b94a-2f1002662d2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.407676 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.407614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:28.543831 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.543802 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v"] Jul 07 17:37:28.546233 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:37:28.546200 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd684d89_b605_4a33_b94a_2f1002662d2a.slice/crio-146391925b4a4a2e45936f8df845af545b5d0fe51e418e310cdfb8756267c1d4 WatchSource:0}: Error finding container 146391925b4a4a2e45936f8df845af545b5d0fe51e418e310cdfb8756267c1d4: Status 404 returned error can't find the container with id 146391925b4a4a2e45936f8df845af545b5d0fe51e418e310cdfb8756267c1d4 Jul 07 17:37:28.547507 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:28.547490 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:37:29.450377 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:29.450323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" event={"ID":"fd684d89-b605-4a33-b94a-2f1002662d2a","Type":"ContainerStarted","Data":"146391925b4a4a2e45936f8df845af545b5d0fe51e418e310cdfb8756267c1d4"} Jul 07 17:37:30.456545 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:30.456510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" event={"ID":"fd684d89-b605-4a33-b94a-2f1002662d2a","Type":"ContainerStarted","Data":"f8f9a619b89bad38452c66b21a9cdc65af7344a6de41657d78e93c0ffe45b053"} Jul 07 17:37:30.485911 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:30.485859 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" podStartSLOduration=1.455256646 podStartE2EDuration="2.485843139s" totalImagesPullingTime="1.030586493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:37:30.484245376 +0000 UTC m=+885.324559047" watchObservedRunningTime="2026-07-07 17:37:30.485843139 +0000 UTC m=+885.326156810" Jul 07 17:37:38.408348 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:38.408260 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:38.408847 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:38.408396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:38.421316 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:38.421289 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:37:38.502246 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:37:38.502214 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v" Jul 07 17:52:49.641802 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.641768 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8"] Jul 07 17:52:49.644577 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.644554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.647685 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.647660 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:52:49.647796 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.647686 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-migrate-9952c532-serving-cert\"" Jul 07 17:52:49.661340 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.661320 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8"] Jul 07 17:52:49.798174 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.798144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8x4\" (UniqueName: \"kubernetes.io/projected/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-kube-api-access-vj8x4\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.798328 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.798183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-maas-api-tls\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.798328 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.798205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.899509 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.899448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-maas-api-tls\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.899509 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.899487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.899631 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.899549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8x4\" (UniqueName: \"kubernetes.io/projected/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-kube-api-access-vj8x4\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.900178 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.900161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.901777 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.901761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-maas-api-tls\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.918149 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.918121 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8x4\" (UniqueName: \"kubernetes.io/projected/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-kube-api-access-vj8x4\") pod \"maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:49.957055 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:49.957027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:50.092735 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:52:50.092703 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aad67bb_23b1_4c31_b542_ffb8a5ce37b7.slice/crio-02fb87fbea3809bf8f3554c18cfda322ebb947f01be4143c5769a984ea7f9cd8 WatchSource:0}: Error finding container 02fb87fbea3809bf8f3554c18cfda322ebb947f01be4143c5769a984ea7f9cd8: Status 404 returned error can't find the container with id 02fb87fbea3809bf8f3554c18cfda322ebb947f01be4143c5769a984ea7f9cd8 Jul 07 17:52:50.094113 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:50.094089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8"] Jul 07 17:52:50.094291 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:50.094252 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:52:50.458206 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:50.458163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" event={"ID":"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7","Type":"ContainerStarted","Data":"02fb87fbea3809bf8f3554c18cfda322ebb947f01be4143c5769a984ea7f9cd8"} Jul 07 17:52:52.469288 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:52.469188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" event={"ID":"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7","Type":"ContainerStarted","Data":"d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6"} Jul 07 17:52:52.469288 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:52.469282 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:52.491084 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:52.491033 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" podStartSLOduration=1.434259658 podStartE2EDuration="3.491020257s" totalImagesPullingTime="2.056760599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:52:52.490213329 +0000 UTC m=+1807.330526999" watchObservedRunningTime="2026-07-07 17:52:52.491020257 +0000 UTC m=+1807.331333928" Jul 07 17:52:54.613000 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:54.612965 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8"] Jul 07 17:52:54.613410 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:54.613169 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" podUID="2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" containerName="maas-api" containerID="cri-o://d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6" gracePeriod=30 Jul 07 17:52:54.862598 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:54.862575 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:55.042345 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.042310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj8x4\" (UniqueName: \"kubernetes.io/projected/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-kube-api-access-vj8x4\") pod \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " Jul 07 17:52:55.042345 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.042351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-maas-api-tls\") pod \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " Jul 07 17:52:55.042595 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.042417 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-openshift-service-ca\") pod \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\" (UID: \"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7\") " Jul 07 17:52:55.042791 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.042766 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" (UID: "2aad67bb-23b1-4c31-b542-ffb8a5ce37b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:52:55.044635 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.044611 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-kube-api-access-vj8x4" (OuterVolumeSpecName: "kube-api-access-vj8x4") pod "2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" (UID: "2aad67bb-23b1-4c31-b542-ffb8a5ce37b7"). InnerVolumeSpecName "kube-api-access-vj8x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:52:55.044721 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.044664 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" (UID: "2aad67bb-23b1-4c31-b542-ffb8a5ce37b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:52:55.143470 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.143436 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8x4\" (UniqueName: \"kubernetes.io/projected/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-kube-api-access-vj8x4\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:52:55.143470 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.143463 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-maas-api-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:52:55.143470 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.143473 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7-openshift-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:52:55.483331 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.483298 2579 generic.go:356] "Generic (PLEG): container finished" podID="2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" containerID="d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6" exitCode=0 Jul 07 17:52:55.483474 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.483339 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" Jul 07 17:52:55.483474 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.483347 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" event={"ID":"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7","Type":"ContainerDied","Data":"d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6"} Jul 07 17:52:55.483474 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.483402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8" event={"ID":"2aad67bb-23b1-4c31-b542-ffb8a5ce37b7","Type":"ContainerDied","Data":"02fb87fbea3809bf8f3554c18cfda322ebb947f01be4143c5769a984ea7f9cd8"} Jul 07 17:52:55.483474 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.483432 2579 scope.go:117] "RemoveContainer" containerID="d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6" Jul 07 17:52:55.492801 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.492777 2579 scope.go:117] "RemoveContainer" containerID="d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6" Jul 07 17:52:55.493084 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:52:55.493064 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6\": container with ID starting with d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6 not found: ID does not exist" containerID="d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6" Jul 07 17:52:55.493156 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.493096 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6"} err="failed to get container status \"d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6\": rpc error: code = NotFound desc = could not find container \"d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6\": container with ID starting with d9e63a59d6db73ec4d88412b6c235913189a01359a22a63a04db7f1931ee40b6 not found: ID does not exist" Jul 07 17:52:55.506820 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.506796 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8"] Jul 07 17:52:55.512121 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.512101 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-migrate-9952c532-7565b4c76b-wmtz8"] Jul 07 17:52:55.843346 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:52:55.843250 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" path="/var/lib/kubelet/pods/2aad67bb-23b1-4c31-b542-ffb8a5ce37b7/volumes" Jul 07 17:53:21.688039 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.688000 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv"] Jul 07 17:53:21.688579 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.688557 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" containerName="maas-api" Jul 07 17:53:21.688653 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.688583 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" containerName="maas-api" Jul 07 17:53:21.688704 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.688677 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aad67bb-23b1-4c31-b542-ffb8a5ce37b7" containerName="maas-api" Jul 07 17:53:21.691530 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.691508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.693946 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.693927 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:53:21.694059 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.693924 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-f27f5913-serving-cert\"" Jul 07 17:53:21.701567 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.701544 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv"] Jul 07 17:53:21.761384 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.761359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6qd\" (UniqueName: \"kubernetes.io/projected/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-kube-api-access-xh6qd\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.761494 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.761395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-maas-api-tls\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.761494 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.761456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-openshift-service-ca\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.862069 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.862005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-openshift-service-ca\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.862069 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.862068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6qd\" (UniqueName: \"kubernetes.io/projected/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-kube-api-access-xh6qd\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.862224 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.862092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-maas-api-tls\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.862740 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.862720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-openshift-service-ca\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.864757 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.864731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-maas-api-tls\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:21.871313 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:21.871288 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6qd\" (UniqueName: \"kubernetes.io/projected/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-kube-api-access-xh6qd\") pod \"maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:22.002394 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:22.002372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:22.145506 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:22.145430 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv"] Jul 07 17:53:22.148460 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:53:22.148430 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028c14c3_a2d6_4d7c_aa68_0f90c6ceeec7.slice/crio-866944ef490410f4f52be00ebe73d855b1aadcf27ed208c7e28b2e4a58612c91 WatchSource:0}: Error finding container 866944ef490410f4f52be00ebe73d855b1aadcf27ed208c7e28b2e4a58612c91: Status 404 returned error can't find the container with id 866944ef490410f4f52be00ebe73d855b1aadcf27ed208c7e28b2e4a58612c91 Jul 07 17:53:22.602048 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:22.602024 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" event={"ID":"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7","Type":"ContainerStarted","Data":"866944ef490410f4f52be00ebe73d855b1aadcf27ed208c7e28b2e4a58612c91"} Jul 07 17:53:23.608047 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:23.608013 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" event={"ID":"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7","Type":"ContainerStarted","Data":"062059d12d29a9774cea61a8b1760aa8e07e85a28e069a25a762030f1192017e"} Jul 07 17:53:23.608437 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:23.608143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:23.632139 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:23.632090 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" podStartSLOduration=2.2380995710000002 podStartE2EDuration="2.632075755s" totalImagesPullingTime="393.976184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:53:23.630613079 +0000 UTC m=+1838.470926752" watchObservedRunningTime="2026-07-07 17:53:23.632075755 +0000 UTC m=+1838.472389427" Jul 07 17:53:26.460022 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.459992 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv"] Jul 07 17:53:26.460537 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.460203 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" podUID="028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" containerName="maas-api" containerID="cri-o://062059d12d29a9774cea61a8b1760aa8e07e85a28e069a25a762030f1192017e" gracePeriod=30 Jul 07 17:53:26.622679 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.622650 2579 generic.go:356] "Generic (PLEG): container finished" podID="028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" containerID="062059d12d29a9774cea61a8b1760aa8e07e85a28e069a25a762030f1192017e" exitCode=0 Jul 07 17:53:26.622818 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.622726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" event={"ID":"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7","Type":"ContainerDied","Data":"062059d12d29a9774cea61a8b1760aa8e07e85a28e069a25a762030f1192017e"} Jul 07 17:53:26.718741 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.718689 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:26.801814 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.801786 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-openshift-service-ca\") pod \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " Jul 07 17:53:26.801930 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.801874 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh6qd\" (UniqueName: \"kubernetes.io/projected/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-kube-api-access-xh6qd\") pod \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " Jul 07 17:53:26.801930 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.801896 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-maas-api-tls\") pod \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\" (UID: \"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7\") " Jul 07 17:53:26.802143 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.802121 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" (UID: "028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:53:26.804032 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.804008 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-kube-api-access-xh6qd" (OuterVolumeSpecName: "kube-api-access-xh6qd") pod "028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" (UID: "028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7"). InnerVolumeSpecName "kube-api-access-xh6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:53:26.804092 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.804044 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" (UID: "028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:53:26.903284 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.903245 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6qd\" (UniqueName: \"kubernetes.io/projected/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-kube-api-access-xh6qd\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:53:26.903365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.903286 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-maas-api-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:53:26.903365 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:26.903298 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7-openshift-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:53:27.629388 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:27.629355 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" Jul 07 17:53:27.629833 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:27.629356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv" event={"ID":"028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7","Type":"ContainerDied","Data":"866944ef490410f4f52be00ebe73d855b1aadcf27ed208c7e28b2e4a58612c91"} Jul 07 17:53:27.629833 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:27.629527 2579 scope.go:117] "RemoveContainer" containerID="062059d12d29a9774cea61a8b1760aa8e07e85a28e069a25a762030f1192017e" Jul 07 17:53:27.663911 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:27.663887 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv"] Jul 07 17:53:27.668827 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:27.668806 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-f27f5913-76b8769b7c-xrkgv"] Jul 07 17:53:27.841831 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:53:27.841806 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" path="/var/lib/kubelet/pods/028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7/volumes" Jul 07 17:56:15.208305 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.208244 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx"] Jul 07 17:56:15.208776 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.208578 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" containerName="maas-api" Jul 07 17:56:15.208776 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.208589 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" containerName="maas-api" Jul 07 17:56:15.208776 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.208664 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="028c14c3-a2d6-4d7c-aa68-0f90c6ceeec7" containerName="maas-api" Jul 07 17:56:15.211710 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.211691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.216032 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.216013 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb345660\"/\"openshift-service-ca.crt\"" Jul 07 17:56:15.216954 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.216935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb345660\"/\"e2e-lifecycle-model-cb345660-kserve-self-signed-certs\"" Jul 07 17:56:15.217054 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.217003 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb345660\"/\"kube-root-ca.crt\"" Jul 07 17:56:15.217125 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.217051 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb345660\"/\"default-dockercfg-k75rj\"" Jul 07 17:56:15.225624 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.225603 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx"] Jul 07 17:56:15.302864 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.302836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgtc\" (UniqueName: \"kubernetes.io/projected/15824e6e-5f74-4068-a0a3-36d6f546bfa3-kube-api-access-fpgtc\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.302982 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.302873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-home\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.302982 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.302898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-model-cache\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.302982 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.302959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tmp-dir\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.303128 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.303009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-dshm\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.303128 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.303047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tls-certs\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.403722 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.403688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgtc\" (UniqueName: \"kubernetes.io/projected/15824e6e-5f74-4068-a0a3-36d6f546bfa3-kube-api-access-fpgtc\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.403722 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.403722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-home\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.403925 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.403746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-model-cache\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.403925 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.403782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tmp-dir\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.403925 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.403840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-dshm\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.403925 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.403876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tls-certs\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.404193 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.404171 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-home\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.404239 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.404180 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-model-cache\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.404336 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.404317 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tmp-dir\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.406201 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.406183 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-dshm\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.406479 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.406461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tls-certs\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.419130 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.419102 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgtc\" (UniqueName: \"kubernetes.io/projected/15824e6e-5f74-4068-a0a3-36d6f546bfa3-kube-api-access-fpgtc\") pod \"e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.520487 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.520450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:15.664385 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:15.664362 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx"] Jul 07 17:56:15.666509 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:56:15.666479 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15824e6e_5f74_4068_a0a3_36d6f546bfa3.slice/crio-473216c0435cf0e5f5cdd4f8ef93f89b4ad5c55fa5e2d857854bfa367a85e354 WatchSource:0}: Error finding container 473216c0435cf0e5f5cdd4f8ef93f89b4ad5c55fa5e2d857854bfa367a85e354: Status 404 returned error can't find the container with id 473216c0435cf0e5f5cdd4f8ef93f89b4ad5c55fa5e2d857854bfa367a85e354 Jul 07 17:56:16.365221 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:16.365187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" event={"ID":"15824e6e-5f74-4068-a0a3-36d6f546bfa3","Type":"ContainerStarted","Data":"97d2be5ec8e384fce58ca38307a86e193f3c6009f9d0655dcc93c197914cc937"} Jul 07 17:56:16.365221 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:16.365227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" event={"ID":"15824e6e-5f74-4068-a0a3-36d6f546bfa3","Type":"ContainerStarted","Data":"473216c0435cf0e5f5cdd4f8ef93f89b4ad5c55fa5e2d857854bfa367a85e354"} Jul 07 17:56:16.400332 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:16.400218 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" podStartSLOduration=1.400198016 podStartE2EDuration="1.400198016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:56:16.397169459 +0000 UTC m=+2011.237483130" watchObservedRunningTime="2026-07-07 17:56:16.400198016 +0000 UTC m=+2011.240511688" Jul 07 17:56:25.520706 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:25.520671 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:25.520706 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:25.520714 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:25.533025 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:25.532997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:26.418380 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:26.418349 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:56:51.404625 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:56:51.404592 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx"] Jul 07 17:57:06.425221 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.425113 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" podUID="15824e6e-5f74-4068-a0a3-36d6f546bfa3" containerName="main" containerID="cri-o://97d2be5ec8e384fce58ca38307a86e193f3c6009f9d0655dcc93c197914cc937" gracePeriod=45 Jul 07 17:57:06.592417 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.592384 2579 generic.go:356] "Generic (PLEG): container finished" podID="15824e6e-5f74-4068-a0a3-36d6f546bfa3" containerID="97d2be5ec8e384fce58ca38307a86e193f3c6009f9d0655dcc93c197914cc937" exitCode=0 Jul 07 17:57:06.592569 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.592452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" event={"ID":"15824e6e-5f74-4068-a0a3-36d6f546bfa3","Type":"ContainerDied","Data":"97d2be5ec8e384fce58ca38307a86e193f3c6009f9d0655dcc93c197914cc937"} Jul 07 17:57:06.675143 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.675116 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:57:06.711187 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711119 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpgtc\" (UniqueName: \"kubernetes.io/projected/15824e6e-5f74-4068-a0a3-36d6f546bfa3-kube-api-access-fpgtc\") pod \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " Jul 07 17:57:06.711187 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711155 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-model-cache\") pod \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " Jul 07 17:57:06.711383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711302 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tmp-dir\") pod \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " Jul 07 17:57:06.711383 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711349 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-home\") pod \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " Jul 07 17:57:06.711498 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711395 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tls-certs\") pod \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " Jul 07 17:57:06.711498 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711422 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-dshm\") pod \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\" (UID: \"15824e6e-5f74-4068-a0a3-36d6f546bfa3\") " Jul 07 17:57:06.711599 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711511 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-model-cache" (OuterVolumeSpecName: "model-cache") pod "15824e6e-5f74-4068-a0a3-36d6f546bfa3" (UID: "15824e6e-5f74-4068-a0a3-36d6f546bfa3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:06.711695 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711668 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "15824e6e-5f74-4068-a0a3-36d6f546bfa3" (UID: "15824e6e-5f74-4068-a0a3-36d6f546bfa3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:06.711749 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711715 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-home" (OuterVolumeSpecName: "home") pod "15824e6e-5f74-4068-a0a3-36d6f546bfa3" (UID: "15824e6e-5f74-4068-a0a3-36d6f546bfa3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:06.711749 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711739 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tmp-dir\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.711842 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.711760 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-model-cache\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.713695 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.713665 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15824e6e-5f74-4068-a0a3-36d6f546bfa3-kube-api-access-fpgtc" (OuterVolumeSpecName: "kube-api-access-fpgtc") pod "15824e6e-5f74-4068-a0a3-36d6f546bfa3" (UID: "15824e6e-5f74-4068-a0a3-36d6f546bfa3"). InnerVolumeSpecName "kube-api-access-fpgtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:57:06.714154 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.714132 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15824e6e-5f74-4068-a0a3-36d6f546bfa3" (UID: "15824e6e-5f74-4068-a0a3-36d6f546bfa3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.714245 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.714153 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-dshm" (OuterVolumeSpecName: "dshm") pod "15824e6e-5f74-4068-a0a3-36d6f546bfa3" (UID: "15824e6e-5f74-4068-a0a3-36d6f546bfa3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:06.812891 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.812862 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-home\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.812891 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.812889 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15824e6e-5f74-4068-a0a3-36d6f546bfa3-tls-certs\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.812891 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.812898 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15824e6e-5f74-4068-a0a3-36d6f546bfa3-dshm\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.813115 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:06.812906 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpgtc\" (UniqueName: \"kubernetes.io/projected/15824e6e-5f74-4068-a0a3-36d6f546bfa3-kube-api-access-fpgtc\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 17:57:07.598600 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:07.598560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" event={"ID":"15824e6e-5f74-4068-a0a3-36d6f546bfa3","Type":"ContainerDied","Data":"473216c0435cf0e5f5cdd4f8ef93f89b4ad5c55fa5e2d857854bfa367a85e354"} Jul 07 17:57:07.599016 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:07.598610 2579 scope.go:117] "RemoveContainer" containerID="97d2be5ec8e384fce58ca38307a86e193f3c6009f9d0655dcc93c197914cc937" Jul 07 17:57:07.599016 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:07.598577 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx" Jul 07 17:57:07.622787 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:07.622760 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx"] Jul 07 17:57:07.625075 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:07.625049 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cb345660/e2e-lifecycle-model-cb345660-kserve-f7d5df487-4jhnx"] Jul 07 17:57:07.843039 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:57:07.843007 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15824e6e-5f74-4068-a0a3-36d6f546bfa3" path="/var/lib/kubelet/pods/15824e6e-5f74-4068-a0a3-36d6f546bfa3/volumes" Jul 07 17:58:46.797312 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.797254 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r"] Jul 07 17:58:46.797895 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.797826 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15824e6e-5f74-4068-a0a3-36d6f546bfa3" containerName="main" Jul 07 17:58:46.797895 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.797850 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15824e6e-5f74-4068-a0a3-36d6f546bfa3" containerName="main" Jul 07 17:58:46.798007 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.797964 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15824e6e-5f74-4068-a0a3-36d6f546bfa3" containerName="main" Jul 07 17:58:46.801386 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.801366 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.806953 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.806933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 17:58:46.807597 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.807583 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-7b6b8d-serving-cert\"" Jul 07 17:58:46.821754 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.821729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d84449-511f-4643-a763-c2f81be9a9c4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.821862 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.821781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwtb\" (UniqueName: \"kubernetes.io/projected/f1d84449-511f-4643-a763-c2f81be9a9c4-kube-api-access-hkwtb\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.821862 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.821810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.822214 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.822190 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r"] Jul 07 17:58:46.922958 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.922934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d84449-511f-4643-a763-c2f81be9a9c4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.923091 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.922968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwtb\" (UniqueName: \"kubernetes.io/projected/f1d84449-511f-4643-a763-c2f81be9a9c4-kube-api-access-hkwtb\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.923091 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.922989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.923091 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:58:46.923079 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-7b6b8d-serving-cert: secret "maas-api-e2e-inf-b-7b6b8d-serving-cert" not found Jul 07 17:58:46.923258 ip-10-0-129-95 kubenswrapper[2579]: E0707 17:58:46.923128 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls podName:f1d84449-511f-4643-a763-c2f81be9a9c4 nodeName:}" failed. No retries permitted until 2026-07-07 17:58:47.423112717 +0000 UTC m=+2162.263426367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls") pod "maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" (UID: "f1d84449-511f-4643-a763-c2f81be9a9c4") : secret "maas-api-e2e-inf-b-7b6b8d-serving-cert" not found Jul 07 17:58:46.923659 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.923634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d84449-511f-4643-a763-c2f81be9a9c4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:46.940396 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:46.940368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwtb\" (UniqueName: \"kubernetes.io/projected/f1d84449-511f-4643-a763-c2f81be9a9c4-kube-api-access-hkwtb\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:47.427334 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:47.427301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:47.430085 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:47.430052 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:47.713523 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:47.713442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:47.862518 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:47.862490 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r"] Jul 07 17:58:47.865427 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:58:47.865393 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d84449_511f_4643_a763_c2f81be9a9c4.slice/crio-50763b40c4b68bdde5ff675a1184e8666a0628d44664c7d5c08ad345b2644b63 WatchSource:0}: Error finding container 50763b40c4b68bdde5ff675a1184e8666a0628d44664c7d5c08ad345b2644b63: Status 404 returned error can't find the container with id 50763b40c4b68bdde5ff675a1184e8666a0628d44664c7d5c08ad345b2644b63 Jul 07 17:58:47.866980 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:47.866959 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:58:48.027367 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:48.027334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" event={"ID":"f1d84449-511f-4643-a763-c2f81be9a9c4","Type":"ContainerStarted","Data":"50763b40c4b68bdde5ff675a1184e8666a0628d44664c7d5c08ad345b2644b63"} Jul 07 17:58:49.033474 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:49.033437 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" event={"ID":"f1d84449-511f-4643-a763-c2f81be9a9c4","Type":"ContainerStarted","Data":"572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea"} Jul 07 17:58:49.033901 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:49.033521 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:58:49.064600 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:49.064559 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" podStartSLOduration=2.696690743 podStartE2EDuration="3.064544159s" totalImagesPullingTime="367.853416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:58:49.063904588 +0000 UTC m=+2163.904218260" watchObservedRunningTime="2026-07-07 17:58:49.064544159 +0000 UTC m=+2163.904857829" Jul 07 17:58:55.043250 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:58:55.043218 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 17:59:13.536770 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.536740 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7"] Jul 07 17:59:13.540425 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.540400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.543952 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.543925 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f629b8\"/\"kube-root-ca.crt\"" Jul 07 17:59:13.545175 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.545151 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f629b8\"/\"default-dockercfg-vs7n6\"" Jul 07 17:59:13.545309 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.545199 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f629b8\"/\"test-model-f629b8-kserve-self-signed-certs\"" Jul 07 17:59:13.545309 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.545237 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f629b8\"/\"openshift-service-ca.crt\"" Jul 07 17:59:13.551882 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.551857 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7"] Jul 07 17:59:13.644431 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.644397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-model-cache\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.644603 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.644454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb65s\" (UniqueName: \"kubernetes.io/projected/6a7a7603-e481-4d7e-90b2-e298ba9777f0-kube-api-access-pb65s\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.644603 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.644532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tls-certs\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.644603 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.644563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-dshm\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.644603 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.644581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tmp-dir\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.644747 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.644647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-home\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.745725 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.745685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb65s\" (UniqueName: \"kubernetes.io/projected/6a7a7603-e481-4d7e-90b2-e298ba9777f0-kube-api-access-pb65s\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.745900 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.745751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tls-certs\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.745900 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.745784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-dshm\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.745900 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.745804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tmp-dir\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.745900 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.745834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-home\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.745900 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.745867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-model-cache\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.746334 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.746309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-home\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.746334 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.746328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tmp-dir\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.746449 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.746372 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-model-cache\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.748158 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.748134 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-dshm\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.748467 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.748448 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tls-certs\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.756597 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.756574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb65s\" (UniqueName: \"kubernetes.io/projected/6a7a7603-e481-4d7e-90b2-e298ba9777f0-kube-api-access-pb65s\") pod \"test-model-f629b8-kserve-7d66c47fb7-x7nj7\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.850309 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.850223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:13.986715 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:13.986691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7"] Jul 07 17:59:13.988878 ip-10-0-129-95 kubenswrapper[2579]: W0707 17:59:13.988853 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a7a7603_e481_4d7e_90b2_e298ba9777f0.slice/crio-700d259a30f18b91c0bbcbb60eb4892b619280c39ba12bd235a3a8020a409c42 WatchSource:0}: Error finding container 700d259a30f18b91c0bbcbb60eb4892b619280c39ba12bd235a3a8020a409c42: Status 404 returned error can't find the container with id 700d259a30f18b91c0bbcbb60eb4892b619280c39ba12bd235a3a8020a409c42 Jul 07 17:59:14.141214 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:14.141130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" event={"ID":"6a7a7603-e481-4d7e-90b2-e298ba9777f0","Type":"ContainerStarted","Data":"176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6"} Jul 07 17:59:14.141214 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:14.141168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" event={"ID":"6a7a7603-e481-4d7e-90b2-e298ba9777f0","Type":"ContainerStarted","Data":"700d259a30f18b91c0bbcbb60eb4892b619280c39ba12bd235a3a8020a409c42"} Jul 07 17:59:14.162393 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:14.162343 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" podStartSLOduration=1.162327992 podStartE2EDuration="1.162327992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:59:14.160098195 +0000 UTC m=+2189.000411867" watchObservedRunningTime="2026-07-07 17:59:14.162327992 +0000 UTC m=+2189.002641662" Jul 07 17:59:23.851044 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:23.851017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:23.851475 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:23.851072 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:23.863221 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:23.863200 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 17:59:24.195360 ip-10-0-129-95 kubenswrapper[2579]: I0707 17:59:24.195255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 18:00:00.157592 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.157560 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d"] Jul 07 18:00:00.161718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.161702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" Jul 07 18:00:00.174404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.174383 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d"] Jul 07 18:00:00.214579 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.214553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxfj\" (UniqueName: \"kubernetes.io/projected/6393ced2-61fb-45cd-b572-3fbd5bf35042-kube-api-access-6fxfj\") pod \"maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d\" (UID: \"6393ced2-61fb-45cd-b572-3fbd5bf35042\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" Jul 07 18:00:00.315657 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.315629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxfj\" (UniqueName: \"kubernetes.io/projected/6393ced2-61fb-45cd-b572-3fbd5bf35042-kube-api-access-6fxfj\") pod \"maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d\" (UID: \"6393ced2-61fb-45cd-b572-3fbd5bf35042\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" Jul 07 18:00:00.329083 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.329061 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxfj\" (UniqueName: \"kubernetes.io/projected/6393ced2-61fb-45cd-b572-3fbd5bf35042-kube-api-access-6fxfj\") pod \"maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d\" (UID: \"6393ced2-61fb-45cd-b572-3fbd5bf35042\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" Jul 07 18:00:00.472318 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.472239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" Jul 07 18:00:00.609588 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.609554 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d"] Jul 07 18:00:00.612863 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:00:00.612833 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6393ced2_61fb_45cd_b572_3fbd5bf35042.slice/crio-3b755c4aa189b78543a428a1918a96d2970cf7eceef5c858f2cc993cda0b3fd1 WatchSource:0}: Error finding container 3b755c4aa189b78543a428a1918a96d2970cf7eceef5c858f2cc993cda0b3fd1: Status 404 returned error can't find the container with id 3b755c4aa189b78543a428a1918a96d2970cf7eceef5c858f2cc993cda0b3fd1 Jul 07 18:00:00.954941 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:00.954908 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7"] Jul 07 18:00:01.349978 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:01.349946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" event={"ID":"6393ced2-61fb-45cd-b572-3fbd5bf35042","Type":"ContainerStarted","Data":"3b755c4aa189b78543a428a1918a96d2970cf7eceef5c858f2cc993cda0b3fd1"} Jul 07 18:00:15.974882 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:15.974789 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" podUID="6a7a7603-e481-4d7e-90b2-e298ba9777f0" containerName="main" containerID="cri-o://176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6" gracePeriod=45 Jul 07 18:00:16.220359 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.220329 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 18:00:16.358791 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.358765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-model-cache\") pod \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " Jul 07 18:00:16.358938 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.358794 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tmp-dir\") pod \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " Jul 07 18:00:16.358938 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.358833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-home\") pod \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " Jul 07 18:00:16.358938 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.358859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tls-certs\") pod \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " Jul 07 18:00:16.358938 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.358891 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-dshm\") pod \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " Jul 07 18:00:16.358938 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.358926 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb65s\" (UniqueName: \"kubernetes.io/projected/6a7a7603-e481-4d7e-90b2-e298ba9777f0-kube-api-access-pb65s\") pod \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\" (UID: \"6a7a7603-e481-4d7e-90b2-e298ba9777f0\") " Jul 07 18:00:16.359181 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.359062 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a7a7603-e481-4d7e-90b2-e298ba9777f0" (UID: "6a7a7603-e481-4d7e-90b2-e298ba9777f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:16.359181 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.359078 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a7a7603-e481-4d7e-90b2-e298ba9777f0" (UID: "6a7a7603-e481-4d7e-90b2-e298ba9777f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:16.359181 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.359121 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-home" (OuterVolumeSpecName: "home") pod "6a7a7603-e481-4d7e-90b2-e298ba9777f0" (UID: "6a7a7603-e481-4d7e-90b2-e298ba9777f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:16.359347 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.359217 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-home\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:16.359347 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.359232 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-model-cache\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:16.359347 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.359246 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tmp-dir\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:16.361220 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.361189 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7a7603-e481-4d7e-90b2-e298ba9777f0-kube-api-access-pb65s" (OuterVolumeSpecName: "kube-api-access-pb65s") pod "6a7a7603-e481-4d7e-90b2-e298ba9777f0" (UID: "6a7a7603-e481-4d7e-90b2-e298ba9777f0"). InnerVolumeSpecName "kube-api-access-pb65s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:00:16.361220 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.361195 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a7a7603-e481-4d7e-90b2-e298ba9777f0" (UID: "6a7a7603-e481-4d7e-90b2-e298ba9777f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:00:16.361220 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.361194 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-dshm" (OuterVolumeSpecName: "dshm") pod "6a7a7603-e481-4d7e-90b2-e298ba9777f0" (UID: "6a7a7603-e481-4d7e-90b2-e298ba9777f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:00:16.415526 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.415505 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a7a7603-e481-4d7e-90b2-e298ba9777f0" containerID="176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6" exitCode=0 Jul 07 18:00:16.415623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.415549 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" Jul 07 18:00:16.415623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.415559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" event={"ID":"6a7a7603-e481-4d7e-90b2-e298ba9777f0","Type":"ContainerDied","Data":"176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6"} Jul 07 18:00:16.415623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.415589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7" event={"ID":"6a7a7603-e481-4d7e-90b2-e298ba9777f0","Type":"ContainerDied","Data":"700d259a30f18b91c0bbcbb60eb4892b619280c39ba12bd235a3a8020a409c42"} Jul 07 18:00:16.415623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.415606 2579 scope.go:117] "RemoveContainer" containerID="176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6" Jul 07 18:00:16.424255 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.424234 2579 scope.go:117] "RemoveContainer" containerID="176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6" Jul 07 18:00:16.424517 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:00:16.424501 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6\": container with ID starting with 176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6 not found: ID does not exist" containerID="176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6" Jul 07 18:00:16.424565 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.424524 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6"} err="failed to get container status \"176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6\": rpc error: code = NotFound desc = could not find container \"176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6\": container with ID starting with 176bbe68c4e6d73481cb5cd76cdcb4ab22f1be87eb9f397c93fce8d0a1ed45b6 not found: ID does not exist" Jul 07 18:00:16.449059 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.449035 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7"] Jul 07 18:00:16.450511 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.450492 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f629b8/test-model-f629b8-kserve-7d66c47fb7-x7nj7"] Jul 07 18:00:16.464856 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.460444 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb65s\" (UniqueName: \"kubernetes.io/projected/6a7a7603-e481-4d7e-90b2-e298ba9777f0-kube-api-access-pb65s\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:16.464856 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.460472 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a7603-e481-4d7e-90b2-e298ba9777f0-tls-certs\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:16.464856 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:16.460489 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7a7603-e481-4d7e-90b2-e298ba9777f0-dshm\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:17.843013 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:17.842978 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7a7603-e481-4d7e-90b2-e298ba9777f0" path="/var/lib/kubelet/pods/6a7a7603-e481-4d7e-90b2-e298ba9777f0/volumes" Jul 07 18:00:23.449462 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:23.449371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" event={"ID":"6393ced2-61fb-45cd-b572-3fbd5bf35042","Type":"ContainerStarted","Data":"d0a45804b5f3da5dbdb4b8e9b72697b25a9e1bf8b0b629f382893826dec3af68"} Jul 07 18:00:23.467761 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:23.467714 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" podStartSLOduration=0.975007251 podStartE2EDuration="23.467700941s" totalImagesPullingTime="22.49269369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:23.466873635 +0000 UTC m=+2258.307187307" watchObservedRunningTime="2026-07-07 18:00:23.467700941 +0000 UTC m=+2258.308014612" Jul 07 18:00:24.455691 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:24.455659 2579 generic.go:356] "Generic (PLEG): container finished" podID="6393ced2-61fb-45cd-b572-3fbd5bf35042" containerID="d0a45804b5f3da5dbdb4b8e9b72697b25a9e1bf8b0b629f382893826dec3af68" exitCode=0 Jul 07 18:00:24.456024 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:24.455712 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" event={"ID":"6393ced2-61fb-45cd-b572-3fbd5bf35042","Type":"ContainerDied","Data":"d0a45804b5f3da5dbdb4b8e9b72697b25a9e1bf8b0b629f382893826dec3af68"} Jul 07 18:00:25.600106 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:25.600079 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" Jul 07 18:00:25.732366 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:25.732258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fxfj\" (UniqueName: \"kubernetes.io/projected/6393ced2-61fb-45cd-b572-3fbd5bf35042-kube-api-access-6fxfj\") pod \"6393ced2-61fb-45cd-b572-3fbd5bf35042\" (UID: \"6393ced2-61fb-45cd-b572-3fbd5bf35042\") " Jul 07 18:00:25.734349 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:25.734321 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6393ced2-61fb-45cd-b572-3fbd5bf35042-kube-api-access-6fxfj" (OuterVolumeSpecName: "kube-api-access-6fxfj") pod "6393ced2-61fb-45cd-b572-3fbd5bf35042" (UID: "6393ced2-61fb-45cd-b572-3fbd5bf35042"). InnerVolumeSpecName "kube-api-access-6fxfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:00:25.833195 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:25.833164 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fxfj\" (UniqueName: \"kubernetes.io/projected/6393ced2-61fb-45cd-b572-3fbd5bf35042-kube-api-access-6fxfj\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:26.465785 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:26.465759 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" Jul 07 18:00:26.465952 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:26.465755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d" event={"ID":"6393ced2-61fb-45cd-b572-3fbd5bf35042","Type":"ContainerDied","Data":"3b755c4aa189b78543a428a1918a96d2970cf7eceef5c858f2cc993cda0b3fd1"} Jul 07 18:00:26.465952 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:26.465877 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b755c4aa189b78543a428a1918a96d2970cf7eceef5c858f2cc993cda0b3fd1" Jul 07 18:00:49.619304 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:49.619252 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r"] Jul 07 18:00:49.619778 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:49.619479 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" podUID="f1d84449-511f-4643-a763-c2f81be9a9c4" containerName="maas-api" containerID="cri-o://572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea" gracePeriod=30 Jul 07 18:00:49.679810 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:49.679779 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d"] Jul 07 18:00:49.682703 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:49.682677 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-7b6b8d-29724120-xc86d"] Jul 07 18:00:49.842953 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:49.842924 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6393ced2-61fb-45cd-b572-3fbd5bf35042" path="/var/lib/kubelet/pods/6393ced2-61fb-45cd-b572-3fbd5bf35042/volumes" Jul 07 18:00:49.852594 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:49.852573 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 18:00:50.026697 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.026667 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d84449-511f-4643-a763-c2f81be9a9c4-openshift-service-ca\") pod \"f1d84449-511f-4643-a763-c2f81be9a9c4\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " Jul 07 18:00:50.026840 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.026732 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkwtb\" (UniqueName: \"kubernetes.io/projected/f1d84449-511f-4643-a763-c2f81be9a9c4-kube-api-access-hkwtb\") pod \"f1d84449-511f-4643-a763-c2f81be9a9c4\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " Jul 07 18:00:50.026840 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.026778 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls\") pod \"f1d84449-511f-4643-a763-c2f81be9a9c4\" (UID: \"f1d84449-511f-4643-a763-c2f81be9a9c4\") " Jul 07 18:00:50.027058 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.027033 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d84449-511f-4643-a763-c2f81be9a9c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1d84449-511f-4643-a763-c2f81be9a9c4" (UID: "f1d84449-511f-4643-a763-c2f81be9a9c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:00:50.028919 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.028895 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d84449-511f-4643-a763-c2f81be9a9c4-kube-api-access-hkwtb" (OuterVolumeSpecName: "kube-api-access-hkwtb") pod "f1d84449-511f-4643-a763-c2f81be9a9c4" (UID: "f1d84449-511f-4643-a763-c2f81be9a9c4"). InnerVolumeSpecName "kube-api-access-hkwtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:00:50.029029 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.028936 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1d84449-511f-4643-a763-c2f81be9a9c4" (UID: "f1d84449-511f-4643-a763-c2f81be9a9c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:00:50.127648 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.127620 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwtb\" (UniqueName: \"kubernetes.io/projected/f1d84449-511f-4643-a763-c2f81be9a9c4-kube-api-access-hkwtb\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:50.127648 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.127643 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d84449-511f-4643-a763-c2f81be9a9c4-maas-api-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:50.127648 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.127652 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d84449-511f-4643-a763-c2f81be9a9c4-openshift-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:00:50.572365 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.572332 2579 generic.go:356] "Generic (PLEG): container finished" podID="f1d84449-511f-4643-a763-c2f81be9a9c4" containerID="572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea" exitCode=0 Jul 07 18:00:50.572516 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.572388 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" Jul 07 18:00:50.572516 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.572390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" event={"ID":"f1d84449-511f-4643-a763-c2f81be9a9c4","Type":"ContainerDied","Data":"572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea"} Jul 07 18:00:50.572637 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.572521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r" event={"ID":"f1d84449-511f-4643-a763-c2f81be9a9c4","Type":"ContainerDied","Data":"50763b40c4b68bdde5ff675a1184e8666a0628d44664c7d5c08ad345b2644b63"} Jul 07 18:00:50.572637 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.572553 2579 scope.go:117] "RemoveContainer" containerID="572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea" Jul 07 18:00:50.581544 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.581526 2579 scope.go:117] "RemoveContainer" containerID="572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea" Jul 07 18:00:50.581815 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:00:50.581796 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea\": container with ID starting with 572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea not found: ID does not exist" containerID="572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea" Jul 07 18:00:50.581863 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.581824 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea"} err="failed to get container status \"572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea\": rpc error: code = NotFound desc = could not find container \"572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea\": container with ID starting with 572a49025c46b25f15a4f69fccec85a3f2b1a24a4caae1934611db938da1b6ea not found: ID does not exist" Jul 07 18:00:50.612978 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.612950 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r"] Jul 07 18:00:50.619918 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:50.619898 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7b6b8d-88f5f6884-wrm7r"] Jul 07 18:00:51.843700 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:00:51.843667 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1d84449-511f-4643-a763-c2f81be9a9c4" path="/var/lib/kubelet/pods/f1d84449-511f-4643-a763-c2f81be9a9c4/volumes" Jul 07 18:01:01.825123 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825095 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f"] Jul 07 18:01:01.825497 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825418 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7a7603-e481-4d7e-90b2-e298ba9777f0" containerName="main" Jul 07 18:01:01.825497 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825430 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7a7603-e481-4d7e-90b2-e298ba9777f0" containerName="main" Jul 07 18:01:01.825497 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825449 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6393ced2-61fb-45cd-b572-3fbd5bf35042" containerName="cleanup" Jul 07 18:01:01.825497 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825455 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6393ced2-61fb-45cd-b572-3fbd5bf35042" containerName="cleanup" Jul 07 18:01:01.825497 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825466 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d84449-511f-4643-a763-c2f81be9a9c4" containerName="maas-api" Jul 07 18:01:01.825497 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825471 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d84449-511f-4643-a763-c2f81be9a9c4" containerName="maas-api" Jul 07 18:01:01.825721 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825521 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7a7603-e481-4d7e-90b2-e298ba9777f0" containerName="main" Jul 07 18:01:01.825721 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825529 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d84449-511f-4643-a763-c2f81be9a9c4" containerName="maas-api" Jul 07 18:01:01.825721 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.825536 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6393ced2-61fb-45cd-b572-3fbd5bf35042" containerName="cleanup" Jul 07 18:01:01.829540 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.829519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:01.833641 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.833617 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-17ec16-serving-cert\"" Jul 07 18:01:01.833767 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.833625 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 18:01:01.920589 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.920553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:01.920789 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.920613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjsj\" (UniqueName: \"kubernetes.io/projected/03175c0f-59a2-400b-9bfc-d16bf43ae94f-kube-api-access-6mjsj\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:01.920789 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.920684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03175c0f-59a2-400b-9bfc-d16bf43ae94f-openshift-service-ca\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:01.969693 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:01.969660 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f"] Jul 07 18:01:02.021621 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.021585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.021766 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.021632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjsj\" (UniqueName: \"kubernetes.io/projected/03175c0f-59a2-400b-9bfc-d16bf43ae94f-kube-api-access-6mjsj\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.021766 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.021655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03175c0f-59a2-400b-9bfc-d16bf43ae94f-openshift-service-ca\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.021766 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:01:02.021732 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-17ec16-serving-cert: secret "maas-api-e2e-api-a-17ec16-serving-cert" not found Jul 07 18:01:02.021876 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:01:02.021803 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls podName:03175c0f-59a2-400b-9bfc-d16bf43ae94f nodeName:}" failed. No retries permitted until 2026-07-07 18:01:02.521787381 +0000 UTC m=+2297.362101031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls") pod "maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" (UID: "03175c0f-59a2-400b-9bfc-d16bf43ae94f") : secret "maas-api-e2e-api-a-17ec16-serving-cert" not found Jul 07 18:01:02.022210 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.022194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03175c0f-59a2-400b-9bfc-d16bf43ae94f-openshift-service-ca\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.041963 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.041926 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjsj\" (UniqueName: \"kubernetes.io/projected/03175c0f-59a2-400b-9bfc-d16bf43ae94f-kube-api-access-6mjsj\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.525498 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.525466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.528107 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.528081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls\") pod \"maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.740493 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.740458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:02.917467 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:02.917437 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f"] Jul 07 18:01:02.920504 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:01:02.920480 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03175c0f_59a2_400b_9bfc_d16bf43ae94f.slice/crio-7b1c155af1b506fa0d9518da99831ee41b48b03de76f494d7fc6bc0856ed5669 WatchSource:0}: Error finding container 7b1c155af1b506fa0d9518da99831ee41b48b03de76f494d7fc6bc0856ed5669: Status 404 returned error can't find the container with id 7b1c155af1b506fa0d9518da99831ee41b48b03de76f494d7fc6bc0856ed5669 Jul 07 18:01:03.632205 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:03.632172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" event={"ID":"03175c0f-59a2-400b-9bfc-d16bf43ae94f","Type":"ContainerStarted","Data":"6284feb0713c51a0fa9c3a821a887b0561d928832d071470b1488604e6c6bfa1"} Jul 07 18:01:03.632387 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:03.632224 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" event={"ID":"03175c0f-59a2-400b-9bfc-d16bf43ae94f","Type":"ContainerStarted","Data":"7b1c155af1b506fa0d9518da99831ee41b48b03de76f494d7fc6bc0856ed5669"} Jul 07 18:01:03.632387 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:03.632259 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:03.661812 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:03.661767 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" podStartSLOduration=2.143630278 podStartE2EDuration="2.661752647s" totalImagesPullingTime="518.122369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:01:03.659108278 +0000 UTC m=+2298.499421949" watchObservedRunningTime="2026-07-07 18:01:03.661752647 +0000 UTC m=+2298.502066318" Jul 07 18:01:09.642024 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:09.641995 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:10.776703 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:10.776669 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk"] Jul 07 18:01:10.779235 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:10.779219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:10.782193 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:10.782168 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-97c1e9-serving-cert\"" Jul 07 18:01:10.814070 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:10.814045 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk"] Jul 07 18:01:10.899788 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:10.899757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:10.899959 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:10.899791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6809f0-e985-4095-aacb-db3de1009799-openshift-service-ca\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:10.899959 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:10.899843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2g5\" (UniqueName: \"kubernetes.io/projected/7d6809f0-e985-4095-aacb-db3de1009799-kube-api-access-4c2g5\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.001067 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.001031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2g5\" (UniqueName: \"kubernetes.io/projected/7d6809f0-e985-4095-aacb-db3de1009799-kube-api-access-4c2g5\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.001198 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.001076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.001198 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.001113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6809f0-e985-4095-aacb-db3de1009799-openshift-service-ca\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.001365 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:01:11.001256 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-97c1e9-serving-cert: secret "maas-api-e2e-api-b-97c1e9-serving-cert" not found Jul 07 18:01:11.001365 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:01:11.001363 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls podName:7d6809f0-e985-4095-aacb-db3de1009799 nodeName:}" failed. No retries permitted until 2026-07-07 18:01:11.501340304 +0000 UTC m=+2306.341653955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls") pod "maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" (UID: "7d6809f0-e985-4095-aacb-db3de1009799") : secret "maas-api-e2e-api-b-97c1e9-serving-cert" not found Jul 07 18:01:11.001780 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.001760 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6809f0-e985-4095-aacb-db3de1009799-openshift-service-ca\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.041846 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.041777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2g5\" (UniqueName: \"kubernetes.io/projected/7d6809f0-e985-4095-aacb-db3de1009799-kube-api-access-4c2g5\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.505093 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.505060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.507644 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.507622 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls\") pod \"maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.690696 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.690665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:11.830855 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:11.830817 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk"] Jul 07 18:01:11.851282 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:01:11.851237 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6809f0_e985_4095_aacb_db3de1009799.slice/crio-2c17a43a35b3fe86aebc51efd276c28e34c7afe5da95a544f4b6737eac126c32 WatchSource:0}: Error finding container 2c17a43a35b3fe86aebc51efd276c28e34c7afe5da95a544f4b6737eac126c32: Status 404 returned error can't find the container with id 2c17a43a35b3fe86aebc51efd276c28e34c7afe5da95a544f4b6737eac126c32 Jul 07 18:01:12.672786 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:12.672706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" event={"ID":"7d6809f0-e985-4095-aacb-db3de1009799","Type":"ContainerStarted","Data":"0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409"} Jul 07 18:01:12.672786 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:12.672758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" event={"ID":"7d6809f0-e985-4095-aacb-db3de1009799","Type":"ContainerStarted","Data":"2c17a43a35b3fe86aebc51efd276c28e34c7afe5da95a544f4b6737eac126c32"} Jul 07 18:01:12.672980 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:12.672798 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:12.711220 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:12.711162 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" podStartSLOduration=2.187499059 podStartE2EDuration="2.711142078s" totalImagesPullingTime="523.643019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:01:12.708983004 +0000 UTC m=+2307.549296681" watchObservedRunningTime="2026-07-07 18:01:12.711142078 +0000 UTC m=+2307.551455750" Jul 07 18:01:18.683002 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:18.682971 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:24.588618 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.588588 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f"] Jul 07 18:01:24.589062 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.588778 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" podUID="03175c0f-59a2-400b-9bfc-d16bf43ae94f" containerName="maas-api" containerID="cri-o://6284feb0713c51a0fa9c3a821a887b0561d928832d071470b1488604e6c6bfa1" gracePeriod=30 Jul 07 18:01:24.636986 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.636944 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" podUID="03175c0f-59a2-400b-9bfc-d16bf43ae94f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.30:8443/health\": dial tcp 10.134.0.30:8443: connect: connection refused" Jul 07 18:01:24.726904 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.726870 2579 generic.go:356] "Generic (PLEG): container finished" podID="03175c0f-59a2-400b-9bfc-d16bf43ae94f" containerID="6284feb0713c51a0fa9c3a821a887b0561d928832d071470b1488604e6c6bfa1" exitCode=0 Jul 07 18:01:24.727042 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.726941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" event={"ID":"03175c0f-59a2-400b-9bfc-d16bf43ae94f","Type":"ContainerDied","Data":"6284feb0713c51a0fa9c3a821a887b0561d928832d071470b1488604e6c6bfa1"} Jul 07 18:01:24.833805 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.833782 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:24.914316 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.914244 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mjsj\" (UniqueName: \"kubernetes.io/projected/03175c0f-59a2-400b-9bfc-d16bf43ae94f-kube-api-access-6mjsj\") pod \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " Jul 07 18:01:24.914442 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.914374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03175c0f-59a2-400b-9bfc-d16bf43ae94f-openshift-service-ca\") pod \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " Jul 07 18:01:24.914442 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.914413 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls\") pod \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\" (UID: \"03175c0f-59a2-400b-9bfc-d16bf43ae94f\") " Jul 07 18:01:24.914704 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.914682 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03175c0f-59a2-400b-9bfc-d16bf43ae94f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03175c0f-59a2-400b-9bfc-d16bf43ae94f" (UID: "03175c0f-59a2-400b-9bfc-d16bf43ae94f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:01:24.916477 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.916457 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03175c0f-59a2-400b-9bfc-d16bf43ae94f" (UID: "03175c0f-59a2-400b-9bfc-d16bf43ae94f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:01:24.916599 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:24.916579 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03175c0f-59a2-400b-9bfc-d16bf43ae94f-kube-api-access-6mjsj" (OuterVolumeSpecName: "kube-api-access-6mjsj") pod "03175c0f-59a2-400b-9bfc-d16bf43ae94f" (UID: "03175c0f-59a2-400b-9bfc-d16bf43ae94f"). InnerVolumeSpecName "kube-api-access-6mjsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:25.016038 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.016008 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03175c0f-59a2-400b-9bfc-d16bf43ae94f-openshift-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:01:25.016038 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.016034 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03175c0f-59a2-400b-9bfc-d16bf43ae94f-maas-api-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:01:25.016188 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.016048 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mjsj\" (UniqueName: \"kubernetes.io/projected/03175c0f-59a2-400b-9bfc-d16bf43ae94f-kube-api-access-6mjsj\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:01:25.732905 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.732867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" event={"ID":"03175c0f-59a2-400b-9bfc-d16bf43ae94f","Type":"ContainerDied","Data":"7b1c155af1b506fa0d9518da99831ee41b48b03de76f494d7fc6bc0856ed5669"} Jul 07 18:01:25.733309 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.732925 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f" Jul 07 18:01:25.733309 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.732973 2579 scope.go:117] "RemoveContainer" containerID="6284feb0713c51a0fa9c3a821a887b0561d928832d071470b1488604e6c6bfa1" Jul 07 18:01:25.759277 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.759242 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f"] Jul 07 18:01:25.767311 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.767288 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-17ec16-58c9bdb4f6-br87f"] Jul 07 18:01:25.841364 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:25.841341 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03175c0f-59a2-400b-9bfc-d16bf43ae94f" path="/var/lib/kubelet/pods/03175c0f-59a2-400b-9bfc-d16bf43ae94f/volumes" Jul 07 18:01:32.847107 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:32.847070 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk"] Jul 07 18:01:32.847575 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:32.847319 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" podUID="7d6809f0-e985-4095-aacb-db3de1009799" containerName="maas-api" containerID="cri-o://0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409" gracePeriod=30 Jul 07 18:01:33.094048 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.094026 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:33.191631 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.191529 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls\") pod \"7d6809f0-e985-4095-aacb-db3de1009799\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " Jul 07 18:01:33.191631 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.191574 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c2g5\" (UniqueName: \"kubernetes.io/projected/7d6809f0-e985-4095-aacb-db3de1009799-kube-api-access-4c2g5\") pod \"7d6809f0-e985-4095-aacb-db3de1009799\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " Jul 07 18:01:33.191876 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.191634 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6809f0-e985-4095-aacb-db3de1009799-openshift-service-ca\") pod \"7d6809f0-e985-4095-aacb-db3de1009799\" (UID: \"7d6809f0-e985-4095-aacb-db3de1009799\") " Jul 07 18:01:33.192117 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.192085 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6809f0-e985-4095-aacb-db3de1009799-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d6809f0-e985-4095-aacb-db3de1009799" (UID: "7d6809f0-e985-4095-aacb-db3de1009799"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:01:33.193919 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.193891 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d6809f0-e985-4095-aacb-db3de1009799" (UID: "7d6809f0-e985-4095-aacb-db3de1009799"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:01:33.194018 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.193933 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6809f0-e985-4095-aacb-db3de1009799-kube-api-access-4c2g5" (OuterVolumeSpecName: "kube-api-access-4c2g5") pod "7d6809f0-e985-4095-aacb-db3de1009799" (UID: "7d6809f0-e985-4095-aacb-db3de1009799"). InnerVolumeSpecName "kube-api-access-4c2g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:01:33.293336 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.293280 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6809f0-e985-4095-aacb-db3de1009799-maas-api-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:01:33.293336 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.293344 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c2g5\" (UniqueName: \"kubernetes.io/projected/7d6809f0-e985-4095-aacb-db3de1009799-kube-api-access-4c2g5\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:01:33.293586 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.293358 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6809f0-e985-4095-aacb-db3de1009799-openshift-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:01:33.773396 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.773355 2579 generic.go:356] "Generic (PLEG): container finished" podID="7d6809f0-e985-4095-aacb-db3de1009799" containerID="0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409" exitCode=0 Jul 07 18:01:33.773577 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.773403 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" Jul 07 18:01:33.773577 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.773441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" event={"ID":"7d6809f0-e985-4095-aacb-db3de1009799","Type":"ContainerDied","Data":"0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409"} Jul 07 18:01:33.773577 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.773496 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk" event={"ID":"7d6809f0-e985-4095-aacb-db3de1009799","Type":"ContainerDied","Data":"2c17a43a35b3fe86aebc51efd276c28e34c7afe5da95a544f4b6737eac126c32"} Jul 07 18:01:33.773577 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.773519 2579 scope.go:117] "RemoveContainer" containerID="0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409" Jul 07 18:01:33.782189 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.782168 2579 scope.go:117] "RemoveContainer" containerID="0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409" Jul 07 18:01:33.782477 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:01:33.782454 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409\": container with ID starting with 0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409 not found: ID does not exist" containerID="0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409" Jul 07 18:01:33.782524 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.782492 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409"} err="failed to get container status \"0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409\": rpc error: code = NotFound desc = could not find container \"0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409\": container with ID starting with 0d93f2a97efa65fb1ac2e2060c34bac811270c7f2d8a279be706044f2df4a409 not found: ID does not exist" Jul 07 18:01:33.831483 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.831461 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk"] Jul 07 18:01:33.834482 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.834459 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-97c1e9-79d5fdfcb6-wwztk"] Jul 07 18:01:33.843019 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:33.842998 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6809f0-e985-4095-aacb-db3de1009799" path="/var/lib/kubelet/pods/7d6809f0-e985-4095-aacb-db3de1009799/volumes" Jul 07 18:01:43.821217 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.821123 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb"] Jul 07 18:01:43.821661 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.821571 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03175c0f-59a2-400b-9bfc-d16bf43ae94f" containerName="maas-api" Jul 07 18:01:43.821661 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.821586 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03175c0f-59a2-400b-9bfc-d16bf43ae94f" containerName="maas-api" Jul 07 18:01:43.821661 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.821596 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6809f0-e985-4095-aacb-db3de1009799" containerName="maas-api" Jul 07 18:01:43.821661 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.821601 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6809f0-e985-4095-aacb-db3de1009799" containerName="maas-api" Jul 07 18:01:43.821661 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.821661 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03175c0f-59a2-400b-9bfc-d16bf43ae94f" containerName="maas-api" Jul 07 18:01:43.821918 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.821670 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6809f0-e985-4095-aacb-db3de1009799" containerName="maas-api" Jul 07 18:01:43.824574 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.824552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.828712 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.828691 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-1e464b-serving-cert\"" Jul 07 18:01:43.828840 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.828720 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rv6qc\"" Jul 07 18:01:43.845615 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.845594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb"] Jul 07 18:01:43.869279 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.869241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.869370 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.869300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.869492 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.869471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt82p\" (UniqueName: \"kubernetes.io/projected/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-kube-api-access-jt82p\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.969996 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.969970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.970108 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.970029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.970173 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:01:43.970160 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-1e464b-serving-cert: secret "maas-api-e2e-shared-a-1e464b-serving-cert" not found Jul 07 18:01:43.970239 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.970212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt82p\" (UniqueName: \"kubernetes.io/projected/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-kube-api-access-jt82p\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.970430 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:01:43.970240 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls podName:856b46a6-b800-4a2e-aed7-d5ca0a62c2d9 nodeName:}" failed. No retries permitted until 2026-07-07 18:01:44.470218695 +0000 UTC m=+2339.310532358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls") pod "maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" (UID: "856b46a6-b800-4a2e-aed7-d5ca0a62c2d9") : secret "maas-api-e2e-shared-a-1e464b-serving-cert" not found Jul 07 18:01:43.970649 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.970632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:43.985938 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:43.985913 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt82p\" (UniqueName: \"kubernetes.io/projected/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-kube-api-access-jt82p\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:44.475141 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:44.475099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:44.477596 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:44.477577 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:44.735896 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:44.735865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:44.871094 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:44.871017 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb"] Jul 07 18:01:44.873639 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:01:44.873614 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856b46a6_b800_4a2e_aed7_d5ca0a62c2d9.slice/crio-8325ea58a69adcfcd4dabe1df8696200a43dccaecce1c907cc0071887848c5da WatchSource:0}: Error finding container 8325ea58a69adcfcd4dabe1df8696200a43dccaecce1c907cc0071887848c5da: Status 404 returned error can't find the container with id 8325ea58a69adcfcd4dabe1df8696200a43dccaecce1c907cc0071887848c5da Jul 07 18:01:45.835297 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:45.835258 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" event={"ID":"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9","Type":"ContainerStarted","Data":"19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610"} Jul 07 18:01:45.835414 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:45.835317 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" event={"ID":"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9","Type":"ContainerStarted","Data":"8325ea58a69adcfcd4dabe1df8696200a43dccaecce1c907cc0071887848c5da"} Jul 07 18:01:45.835414 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:45.835404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:45.873955 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:45.873905 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" podStartSLOduration=2.479962134 podStartE2EDuration="2.873886888s" totalImagesPullingTime="393.924754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:01:45.872346735 +0000 UTC m=+2340.712660419" watchObservedRunningTime="2026-07-07 18:01:45.873886888 +0000 UTC m=+2340.714200559" Jul 07 18:01:51.845135 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:51.845108 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:01:52.925862 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:52.925834 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf"] Jul 07 18:01:52.928093 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:52.928076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:52.931395 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:52.931376 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-df746e-serving-cert\"" Jul 07 18:01:52.941569 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:52.941549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf"] Jul 07 18:01:53.040345 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.040321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jgrp\" (UniqueName: \"kubernetes.io/projected/6797a991-1dfa-4cff-bef6-3d9be7371b3b-kube-api-access-9jgrp\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.040464 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.040405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6797a991-1dfa-4cff-bef6-3d9be7371b3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.040464 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.040428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6797a991-1dfa-4cff-bef6-3d9be7371b3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.141769 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.141745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6797a991-1dfa-4cff-bef6-3d9be7371b3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.141876 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.141780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6797a991-1dfa-4cff-bef6-3d9be7371b3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.141876 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.141816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jgrp\" (UniqueName: \"kubernetes.io/projected/6797a991-1dfa-4cff-bef6-3d9be7371b3b-kube-api-access-9jgrp\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.142448 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.142426 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6797a991-1dfa-4cff-bef6-3d9be7371b3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.144398 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.144380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6797a991-1dfa-4cff-bef6-3d9be7371b3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.150850 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.150826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jgrp\" (UniqueName: \"kubernetes.io/projected/6797a991-1dfa-4cff-bef6-3d9be7371b3b-kube-api-access-9jgrp\") pod \"maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.237770 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.237748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:53.388417 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.388387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf"] Jul 07 18:01:53.391204 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:01:53.391177 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6797a991_1dfa_4cff_bef6_3d9be7371b3b.slice/crio-9beb9b6dc01fe2772f078503801e54942fbc805e83b05ca8f1862694fdcf1ced WatchSource:0}: Error finding container 9beb9b6dc01fe2772f078503801e54942fbc805e83b05ca8f1862694fdcf1ced: Status 404 returned error can't find the container with id 9beb9b6dc01fe2772f078503801e54942fbc805e83b05ca8f1862694fdcf1ced Jul 07 18:01:53.869618 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:53.869587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" event={"ID":"6797a991-1dfa-4cff-bef6-3d9be7371b3b","Type":"ContainerStarted","Data":"9beb9b6dc01fe2772f078503801e54942fbc805e83b05ca8f1862694fdcf1ced"} Jul 07 18:01:54.875662 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:54.875628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" event={"ID":"6797a991-1dfa-4cff-bef6-3d9be7371b3b","Type":"ContainerStarted","Data":"508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145"} Jul 07 18:01:54.876160 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:54.875739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:01:54.898115 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:01:54.898055 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" podStartSLOduration=2.478076725 podStartE2EDuration="2.898037619s" totalImagesPullingTime="419.960894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:01:54.896329777 +0000 UTC m=+2349.736643449" watchObservedRunningTime="2026-07-07 18:01:54.898037619 +0000 UTC m=+2349.738351293" Jul 07 18:02:00.887005 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:00.886969 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:02:48.258814 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.258782 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf"] Jul 07 18:02:48.261004 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.260988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.267578 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.267557 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"kube-root-ca.crt\"" Jul 07 18:02:48.268057 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.268041 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"openshift-service-ca.crt\"" Jul 07 18:02:48.268120 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.268041 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"default-dockercfg-k2r8l\"" Jul 07 18:02:48.286563 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.286541 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-df746e\"/\"auth-test-model-df746e-kserve-self-signed-certs\"" Jul 07 18:02:48.288774 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.288754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf"] Jul 07 18:02:48.383524 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.383502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-dshm\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.383624 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.383528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7s2q\" (UniqueName: \"kubernetes.io/projected/7bdcdd92-d651-4477-a63f-0c292297280d-kube-api-access-r7s2q\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.383624 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.383579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-model-cache\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.383624 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.383595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-home\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.383624 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.383609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-tmp-dir\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.383756 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.383670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcdd92-d651-4477-a63f-0c292297280d-tls-certs\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484376 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcdd92-d651-4477-a63f-0c292297280d-tls-certs\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484475 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-dshm\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484475 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s2q\" (UniqueName: \"kubernetes.io/projected/7bdcdd92-d651-4477-a63f-0c292297280d-kube-api-access-r7s2q\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484475 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-model-cache\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484613 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-home\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484613 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-tmp-dir\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484875 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-model-cache\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484989 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484868 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-tmp-dir\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.484989 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.484910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-home\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.486737 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.486720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-dshm\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.487003 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.486987 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcdd92-d651-4477-a63f-0c292297280d-tls-certs\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.498777 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.498751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s2q\" (UniqueName: \"kubernetes.io/projected/7bdcdd92-d651-4477-a63f-0c292297280d-kube-api-access-r7s2q\") pod \"auth-test-model-df746e-kserve-7c9899d9f-5m9qf\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.569924 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.569864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:48.727055 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:48.727034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf"] Jul 07 18:02:48.728448 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:02:48.728417 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bdcdd92_d651_4477_a63f_0c292297280d.slice/crio-fd6ef6e8dc2a6ce3ff348a3f39cd2671309f1addcbdca3c609a9cf991ccf21ec WatchSource:0}: Error finding container fd6ef6e8dc2a6ce3ff348a3f39cd2671309f1addcbdca3c609a9cf991ccf21ec: Status 404 returned error can't find the container with id fd6ef6e8dc2a6ce3ff348a3f39cd2671309f1addcbdca3c609a9cf991ccf21ec Jul 07 18:02:49.116136 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:49.116097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" event={"ID":"7bdcdd92-d651-4477-a63f-0c292297280d","Type":"ContainerStarted","Data":"b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34"} Jul 07 18:02:49.116136 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:49.116137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" event={"ID":"7bdcdd92-d651-4477-a63f-0c292297280d","Type":"ContainerStarted","Data":"fd6ef6e8dc2a6ce3ff348a3f39cd2671309f1addcbdca3c609a9cf991ccf21ec"} Jul 07 18:02:49.146976 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:49.146924 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" podStartSLOduration=1.146907264 podStartE2EDuration="1.146907264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:02:49.144046564 +0000 UTC m=+2403.984360235" watchObservedRunningTime="2026-07-07 18:02:49.146907264 +0000 UTC m=+2403.987220935" Jul 07 18:02:58.570337 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:58.570305 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:58.570808 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:58.570358 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:58.582661 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:58.582638 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:02:59.169312 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:02:59.169285 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:05:00.212444 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:00.212390 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf"] Jul 07 18:05:14.190763 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.190726 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5"] Jul 07 18:05:14.194623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.194606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.198323 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.198297 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"sub-test-model-1e464b-kserve-self-signed-certs\"" Jul 07 18:05:14.198451 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.198325 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"openshift-service-ca.crt\"" Jul 07 18:05:14.198451 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.198330 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"kube-root-ca.crt\"" Jul 07 18:05:14.198451 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.198302 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"default-dockercfg-xjxs7\"" Jul 07 18:05:14.206905 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.206881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5"] Jul 07 18:05:14.295496 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.295468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-tmp-dir\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.295496 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.295502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ff7bd-9213-4e20-a925-3400ca8d138a-tls-certs\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.295697 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.295583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-home\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.295697 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.295605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-model-cache\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.295697 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.295658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-784ft\" (UniqueName: \"kubernetes.io/projected/405ff7bd-9213-4e20-a925-3400ca8d138a-kube-api-access-784ft\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.295697 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.295690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-dshm\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396471 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-tmp-dir\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396471 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ff7bd-9213-4e20-a925-3400ca8d138a-tls-certs\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396696 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-home\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396696 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-model-cache\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396696 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784ft\" (UniqueName: \"kubernetes.io/projected/405ff7bd-9213-4e20-a925-3400ca8d138a-kube-api-access-784ft\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396696 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-dshm\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396908 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396805 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-tmp-dir\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.396908 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-home\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.397004 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.396974 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-model-cache\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.398925 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.398900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-dshm\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.399182 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.399166 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ff7bd-9213-4e20-a925-3400ca8d138a-tls-certs\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.406736 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.406706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-784ft\" (UniqueName: \"kubernetes.io/projected/405ff7bd-9213-4e20-a925-3400ca8d138a-kube-api-access-784ft\") pod \"sub-test-model-1e464b-kserve-6b66fc6987-b97f5\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.504484 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.504448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:14.642116 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.642094 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5"] Jul 07 18:05:14.644419 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:05:14.644385 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod405ff7bd_9213_4e20_a925_3400ca8d138a.slice/crio-dd3f8fba3935dd6abca1dbbea2b9073dee4262237366166001ccecfc1ecd025f WatchSource:0}: Error finding container dd3f8fba3935dd6abca1dbbea2b9073dee4262237366166001ccecfc1ecd025f: Status 404 returned error can't find the container with id dd3f8fba3935dd6abca1dbbea2b9073dee4262237366166001ccecfc1ecd025f Jul 07 18:05:14.771329 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.771228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" event={"ID":"405ff7bd-9213-4e20-a925-3400ca8d138a","Type":"ContainerStarted","Data":"f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2"} Jul 07 18:05:14.771329 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.771297 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" event={"ID":"405ff7bd-9213-4e20-a925-3400ca8d138a","Type":"ContainerStarted","Data":"dd3f8fba3935dd6abca1dbbea2b9073dee4262237366166001ccecfc1ecd025f"} Jul 07 18:05:14.791986 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:14.791932 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" podStartSLOduration=0.791918217 podStartE2EDuration="791.918217ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:05:14.79012571 +0000 UTC m=+2549.630439382" watchObservedRunningTime="2026-07-07 18:05:14.791918217 +0000 UTC m=+2549.632231889" Jul 07 18:05:15.232707 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.232589 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" podUID="7bdcdd92-d651-4477-a63f-0c292297280d" containerName="main" containerID="cri-o://b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34" gracePeriod=45 Jul 07 18:05:15.476000 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.475974 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:05:15.608909 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.608859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-home\") pod \"7bdcdd92-d651-4477-a63f-0c292297280d\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " Jul 07 18:05:15.609080 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.608930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-model-cache\") pod \"7bdcdd92-d651-4477-a63f-0c292297280d\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " Jul 07 18:05:15.609080 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.608959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7s2q\" (UniqueName: \"kubernetes.io/projected/7bdcdd92-d651-4477-a63f-0c292297280d-kube-api-access-r7s2q\") pod \"7bdcdd92-d651-4477-a63f-0c292297280d\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " Jul 07 18:05:15.609080 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609018 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-dshm\") pod \"7bdcdd92-d651-4477-a63f-0c292297280d\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " Jul 07 18:05:15.609080 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609045 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-tmp-dir\") pod \"7bdcdd92-d651-4477-a63f-0c292297280d\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " Jul 07 18:05:15.609241 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609099 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcdd92-d651-4477-a63f-0c292297280d-tls-certs\") pod \"7bdcdd92-d651-4477-a63f-0c292297280d\" (UID: \"7bdcdd92-d651-4477-a63f-0c292297280d\") " Jul 07 18:05:15.609241 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609211 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-home" (OuterVolumeSpecName: "home") pod "7bdcdd92-d651-4477-a63f-0c292297280d" (UID: "7bdcdd92-d651-4477-a63f-0c292297280d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:15.609385 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609230 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bdcdd92-d651-4477-a63f-0c292297280d" (UID: "7bdcdd92-d651-4477-a63f-0c292297280d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:15.609385 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609311 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bdcdd92-d651-4477-a63f-0c292297280d" (UID: "7bdcdd92-d651-4477-a63f-0c292297280d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:15.609489 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609464 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-home\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:05:15.609489 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609478 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-model-cache\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:05:15.609573 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.609492 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-tmp-dir\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:05:15.611416 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.611391 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-dshm" (OuterVolumeSpecName: "dshm") pod "7bdcdd92-d651-4477-a63f-0c292297280d" (UID: "7bdcdd92-d651-4477-a63f-0c292297280d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:05:15.611520 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.611473 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdcdd92-d651-4477-a63f-0c292297280d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bdcdd92-d651-4477-a63f-0c292297280d" (UID: "7bdcdd92-d651-4477-a63f-0c292297280d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:05:15.611562 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.611532 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdcdd92-d651-4477-a63f-0c292297280d-kube-api-access-r7s2q" (OuterVolumeSpecName: "kube-api-access-r7s2q") pod "7bdcdd92-d651-4477-a63f-0c292297280d" (UID: "7bdcdd92-d651-4477-a63f-0c292297280d"). InnerVolumeSpecName "kube-api-access-r7s2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:05:15.710122 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.710086 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcdd92-d651-4477-a63f-0c292297280d-tls-certs\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:05:15.710122 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.710117 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7s2q\" (UniqueName: \"kubernetes.io/projected/7bdcdd92-d651-4477-a63f-0c292297280d-kube-api-access-r7s2q\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:05:15.710122 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.710126 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcdd92-d651-4477-a63f-0c292297280d-dshm\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:05:15.777043 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.777007 2579 generic.go:356] "Generic (PLEG): container finished" podID="7bdcdd92-d651-4477-a63f-0c292297280d" containerID="b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34" exitCode=0 Jul 07 18:05:15.777192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.777051 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" Jul 07 18:05:15.777192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.777085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" event={"ID":"7bdcdd92-d651-4477-a63f-0c292297280d","Type":"ContainerDied","Data":"b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34"} Jul 07 18:05:15.777192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.777127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf" event={"ID":"7bdcdd92-d651-4477-a63f-0c292297280d","Type":"ContainerDied","Data":"fd6ef6e8dc2a6ce3ff348a3f39cd2671309f1addcbdca3c609a9cf991ccf21ec"} Jul 07 18:05:15.777192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.777143 2579 scope.go:117] "RemoveContainer" containerID="b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34" Jul 07 18:05:15.785855 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.785837 2579 scope.go:117] "RemoveContainer" containerID="b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34" Jul 07 18:05:15.786302 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:05:15.786120 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34\": container with ID starting with b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34 not found: ID does not exist" containerID="b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34" Jul 07 18:05:15.786302 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.786145 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34"} err="failed to get container status \"b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34\": rpc error: code = NotFound desc = could not find container \"b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34\": container with ID starting with b85eda6d43e2eeaab6623c930613d4a1a7839301d385673fa8558abf2363dd34 not found: ID does not exist" Jul 07 18:05:15.804411 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.804389 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf"] Jul 07 18:05:15.810847 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.810822 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-df746e/auth-test-model-df746e-kserve-7c9899d9f-5m9qf"] Jul 07 18:05:15.842732 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:15.842711 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bdcdd92-d651-4477-a63f-0c292297280d" path="/var/lib/kubelet/pods/7bdcdd92-d651-4477-a63f-0c292297280d/volumes" Jul 07 18:05:24.504918 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:24.504887 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:24.505327 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:24.504939 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:24.517168 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:24.517146 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:05:24.829206 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:05:24.829141 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:06:20.524720 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:20.524683 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5"] Jul 07 18:06:25.092427 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.092387 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f"] Jul 07 18:06:25.093012 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.092992 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdcdd92-d651-4477-a63f-0c292297280d" containerName="main" Jul 07 18:06:25.093064 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.093016 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdcdd92-d651-4477-a63f-0c292297280d" containerName="main" Jul 07 18:06:25.093130 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.093118 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bdcdd92-d651-4477-a63f-0c292297280d" containerName="main" Jul 07 18:06:25.096192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.096173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.099163 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.099136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e464b\"/\"rate-test-model-1e464b-kserve-self-signed-certs\"" Jul 07 18:06:25.105068 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.105045 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f"] Jul 07 18:06:25.177514 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.177492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckpg\" (UniqueName: \"kubernetes.io/projected/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-kube-api-access-8ckpg\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.177623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.177555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-model-cache\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.177623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.177592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tmp-dir\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.177623 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.177617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tls-certs\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.177726 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.177644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-dshm\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.177726 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.177680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-home\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.278981 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.278945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckpg\" (UniqueName: \"kubernetes.io/projected/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-kube-api-access-8ckpg\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279087 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-model-cache\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279087 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tmp-dir\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279160 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tls-certs\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279245 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-dshm\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279313 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-home\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279491 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279478 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tmp-dir\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279555 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279535 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-model-cache\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.279767 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.279555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-home\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.281663 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.281640 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-dshm\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.281819 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.281803 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tls-certs\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.288325 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.288303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckpg\" (UniqueName: \"kubernetes.io/projected/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-kube-api-access-8ckpg\") pod \"rate-test-model-1e464b-kserve-59cdf77899-dq52f\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.406509 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.406438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:25.540242 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:25.540218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f"] Jul 07 18:06:25.542549 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:06:25.542518 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cce29dd_a243_4d8b_ae94_1af8d216ebaf.slice/crio-6057cbbbdcf5dd8a5b4663e9b19438908528c754dea02cccba1e6a55ad8e092f WatchSource:0}: Error finding container 6057cbbbdcf5dd8a5b4663e9b19438908528c754dea02cccba1e6a55ad8e092f: Status 404 returned error can't find the container with id 6057cbbbdcf5dd8a5b4663e9b19438908528c754dea02cccba1e6a55ad8e092f Jul 07 18:06:26.101410 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:26.101373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" event={"ID":"7cce29dd-a243-4d8b-ae94-1af8d216ebaf","Type":"ContainerStarted","Data":"728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f"} Jul 07 18:06:26.101410 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:26.101411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" event={"ID":"7cce29dd-a243-4d8b-ae94-1af8d216ebaf","Type":"ContainerStarted","Data":"6057cbbbdcf5dd8a5b4663e9b19438908528c754dea02cccba1e6a55ad8e092f"} Jul 07 18:06:26.135531 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:26.135482 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" podStartSLOduration=1.135467626 podStartE2EDuration="1.135467626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:06:26.133991202 +0000 UTC m=+2620.974304873" watchObservedRunningTime="2026-07-07 18:06:26.135467626 +0000 UTC m=+2620.975781296" Jul 07 18:06:35.406751 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.406711 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:35.406751 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.406751 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:35.419171 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.419143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:35.544747 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.544697 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" podUID="405ff7bd-9213-4e20-a925-3400ca8d138a" containerName="main" containerID="cri-o://f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2" gracePeriod=45 Jul 07 18:06:35.792261 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.792229 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:06:35.874869 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.874845 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-model-cache\") pod \"405ff7bd-9213-4e20-a925-3400ca8d138a\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " Jul 07 18:06:35.875015 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.874881 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-home\") pod \"405ff7bd-9213-4e20-a925-3400ca8d138a\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " Jul 07 18:06:35.875015 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.874904 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-tmp-dir\") pod \"405ff7bd-9213-4e20-a925-3400ca8d138a\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " Jul 07 18:06:35.875015 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.874940 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-784ft\" (UniqueName: \"kubernetes.io/projected/405ff7bd-9213-4e20-a925-3400ca8d138a-kube-api-access-784ft\") pod \"405ff7bd-9213-4e20-a925-3400ca8d138a\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " Jul 07 18:06:35.875015 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.874964 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ff7bd-9213-4e20-a925-3400ca8d138a-tls-certs\") pod \"405ff7bd-9213-4e20-a925-3400ca8d138a\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " Jul 07 18:06:35.875015 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.874981 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-dshm\") pod \"405ff7bd-9213-4e20-a925-3400ca8d138a\" (UID: \"405ff7bd-9213-4e20-a925-3400ca8d138a\") " Jul 07 18:06:35.875305 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.875122 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-home" (OuterVolumeSpecName: "home") pod "405ff7bd-9213-4e20-a925-3400ca8d138a" (UID: "405ff7bd-9213-4e20-a925-3400ca8d138a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:35.875305 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.875129 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-model-cache" (OuterVolumeSpecName: "model-cache") pod "405ff7bd-9213-4e20-a925-3400ca8d138a" (UID: "405ff7bd-9213-4e20-a925-3400ca8d138a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:35.875305 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.875232 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "405ff7bd-9213-4e20-a925-3400ca8d138a" (UID: "405ff7bd-9213-4e20-a925-3400ca8d138a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:35.875452 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.875312 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-model-cache\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:06:35.875452 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.875330 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-home\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:06:35.877360 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.877331 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405ff7bd-9213-4e20-a925-3400ca8d138a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "405ff7bd-9213-4e20-a925-3400ca8d138a" (UID: "405ff7bd-9213-4e20-a925-3400ca8d138a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:06:35.877360 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.877345 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405ff7bd-9213-4e20-a925-3400ca8d138a-kube-api-access-784ft" (OuterVolumeSpecName: "kube-api-access-784ft") pod "405ff7bd-9213-4e20-a925-3400ca8d138a" (UID: "405ff7bd-9213-4e20-a925-3400ca8d138a"). InnerVolumeSpecName "kube-api-access-784ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:06:35.877360 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.877350 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-dshm" (OuterVolumeSpecName: "dshm") pod "405ff7bd-9213-4e20-a925-3400ca8d138a" (UID: "405ff7bd-9213-4e20-a925-3400ca8d138a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:06:35.976089 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.976034 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-tmp-dir\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:06:35.976089 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.976054 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-784ft\" (UniqueName: \"kubernetes.io/projected/405ff7bd-9213-4e20-a925-3400ca8d138a-kube-api-access-784ft\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:06:35.976089 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.976063 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405ff7bd-9213-4e20-a925-3400ca8d138a-tls-certs\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:06:35.976089 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:35.976075 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405ff7bd-9213-4e20-a925-3400ca8d138a-dshm\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:06:36.147276 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.147229 2579 generic.go:356] "Generic (PLEG): container finished" podID="405ff7bd-9213-4e20-a925-3400ca8d138a" containerID="f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2" exitCode=0 Jul 07 18:06:36.147417 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.147289 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" Jul 07 18:06:36.147417 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.147320 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" event={"ID":"405ff7bd-9213-4e20-a925-3400ca8d138a","Type":"ContainerDied","Data":"f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2"} Jul 07 18:06:36.147417 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.147365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5" event={"ID":"405ff7bd-9213-4e20-a925-3400ca8d138a","Type":"ContainerDied","Data":"dd3f8fba3935dd6abca1dbbea2b9073dee4262237366166001ccecfc1ecd025f"} Jul 07 18:06:36.147417 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.147382 2579 scope.go:117] "RemoveContainer" containerID="f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2" Jul 07 18:06:36.156328 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.156303 2579 scope.go:117] "RemoveContainer" containerID="f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2" Jul 07 18:06:36.156580 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:06:36.156560 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2\": container with ID starting with f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2 not found: ID does not exist" containerID="f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2" Jul 07 18:06:36.156669 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.156589 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2"} err="failed to get container status \"f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2\": rpc error: code = NotFound desc = could not find container \"f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2\": container with ID starting with f26fb1763024c5fa8f08461cd9a4e2a8fcb7e97d4424aa34f58ec9932cc36dc2 not found: ID does not exist" Jul 07 18:06:36.159646 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.159624 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:06:36.171402 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.171381 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5"] Jul 07 18:06:36.178125 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:36.178104 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/sub-test-model-1e464b-kserve-6b66fc6987-b97f5"] Jul 07 18:06:37.842720 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:37.842685 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="405ff7bd-9213-4e20-a925-3400ca8d138a" path="/var/lib/kubelet/pods/405ff7bd-9213-4e20-a925-3400ca8d138a/volumes" Jul 07 18:06:45.855618 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:06:45.855591 2579 scope.go:117] "RemoveContainer" containerID="d0a45804b5f3da5dbdb4b8e9b72697b25a9e1bf8b0b629f382893826dec3af68" Jul 07 18:07:46.099321 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:46.099222 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f"] Jul 07 18:07:49.034855 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.034825 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb"] Jul 07 18:07:49.037045 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.035084 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" podUID="856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" containerName="maas-api" containerID="cri-o://19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610" gracePeriod=30 Jul 07 18:07:49.275627 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.275600 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:07:49.376557 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.376492 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls\") pod \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " Jul 07 18:07:49.376557 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.376529 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt82p\" (UniqueName: \"kubernetes.io/projected/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-kube-api-access-jt82p\") pod \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " Jul 07 18:07:49.376557 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.376549 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-openshift-service-ca\") pod \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\" (UID: \"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9\") " Jul 07 18:07:49.376957 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.376935 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" (UID: "856b46a6-b800-4a2e-aed7-d5ca0a62c2d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:07:49.378685 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.378665 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-kube-api-access-jt82p" (OuterVolumeSpecName: "kube-api-access-jt82p") pod "856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" (UID: "856b46a6-b800-4a2e-aed7-d5ca0a62c2d9"). InnerVolumeSpecName "kube-api-access-jt82p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:07:49.378750 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.378704 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" (UID: "856b46a6-b800-4a2e-aed7-d5ca0a62c2d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:07:49.478121 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.478095 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-maas-api-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:07:49.478121 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.478117 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt82p\" (UniqueName: \"kubernetes.io/projected/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-kube-api-access-jt82p\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:07:49.478245 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.478127 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9-openshift-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:07:49.485778 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.485754 2579 generic.go:356] "Generic (PLEG): container finished" podID="856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" containerID="19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610" exitCode=0 Jul 07 18:07:49.485862 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.485801 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" Jul 07 18:07:49.485862 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.485843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" event={"ID":"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9","Type":"ContainerDied","Data":"19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610"} Jul 07 18:07:49.485939 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.485911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb" event={"ID":"856b46a6-b800-4a2e-aed7-d5ca0a62c2d9","Type":"ContainerDied","Data":"8325ea58a69adcfcd4dabe1df8696200a43dccaecce1c907cc0071887848c5da"} Jul 07 18:07:49.485939 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.485932 2579 scope.go:117] "RemoveContainer" containerID="19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610" Jul 07 18:07:49.494543 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.494527 2579 scope.go:117] "RemoveContainer" containerID="19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610" Jul 07 18:07:49.494803 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:07:49.494781 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610\": container with ID starting with 19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610 not found: ID does not exist" containerID="19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610" Jul 07 18:07:49.495033 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.494811 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610"} err="failed to get container status \"19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610\": rpc error: code = NotFound desc = could not find container \"19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610\": container with ID starting with 19c96dc42213c62e09896dc6b7ac345e99c584655a2cf3c658d4276fc8461610 not found: ID does not exist" Jul 07 18:07:49.510135 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.510115 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb"] Jul 07 18:07:49.514342 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.514322 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1e464b-6594fbfd4b-vtlbb"] Jul 07 18:07:49.842070 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:07:49.842044 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" path="/var/lib/kubelet/pods/856b46a6-b800-4a2e-aed7-d5ca0a62c2d9/volumes" Jul 07 18:08:01.119224 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.119178 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" podUID="7cce29dd-a243-4d8b-ae94-1af8d216ebaf" containerName="main" containerID="cri-o://728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f" gracePeriod=45 Jul 07 18:08:01.365756 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.365733 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:08:01.473494 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473425 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-dshm\") pod \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " Jul 07 18:08:01.473494 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473469 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-home\") pod \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " Jul 07 18:08:01.473665 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473510 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tls-certs\") pod \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " Jul 07 18:08:01.473665 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473529 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-model-cache\") pod \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " Jul 07 18:08:01.473665 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473548 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tmp-dir\") pod \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " Jul 07 18:08:01.473665 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473602 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ckpg\" (UniqueName: \"kubernetes.io/projected/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-kube-api-access-8ckpg\") pod \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\" (UID: \"7cce29dd-a243-4d8b-ae94-1af8d216ebaf\") " Jul 07 18:08:01.473877 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473816 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-home" (OuterVolumeSpecName: "home") pod "7cce29dd-a243-4d8b-ae94-1af8d216ebaf" (UID: "7cce29dd-a243-4d8b-ae94-1af8d216ebaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:01.473877 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473838 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "7cce29dd-a243-4d8b-ae94-1af8d216ebaf" (UID: "7cce29dd-a243-4d8b-ae94-1af8d216ebaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:01.473961 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.473873 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7cce29dd-a243-4d8b-ae94-1af8d216ebaf" (UID: "7cce29dd-a243-4d8b-ae94-1af8d216ebaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:01.475791 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.475769 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-dshm" (OuterVolumeSpecName: "dshm") pod "7cce29dd-a243-4d8b-ae94-1af8d216ebaf" (UID: "7cce29dd-a243-4d8b-ae94-1af8d216ebaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:08:01.475791 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.475776 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-kube-api-access-8ckpg" (OuterVolumeSpecName: "kube-api-access-8ckpg") pod "7cce29dd-a243-4d8b-ae94-1af8d216ebaf" (UID: "7cce29dd-a243-4d8b-ae94-1af8d216ebaf"). InnerVolumeSpecName "kube-api-access-8ckpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:08:01.475928 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.475772 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7cce29dd-a243-4d8b-ae94-1af8d216ebaf" (UID: "7cce29dd-a243-4d8b-ae94-1af8d216ebaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:08:01.544373 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.544345 2579 generic.go:356] "Generic (PLEG): container finished" podID="7cce29dd-a243-4d8b-ae94-1af8d216ebaf" containerID="728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f" exitCode=0 Jul 07 18:08:01.544482 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.544392 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" Jul 07 18:08:01.544482 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.544416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" event={"ID":"7cce29dd-a243-4d8b-ae94-1af8d216ebaf","Type":"ContainerDied","Data":"728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f"} Jul 07 18:08:01.544482 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.544449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f" event={"ID":"7cce29dd-a243-4d8b-ae94-1af8d216ebaf","Type":"ContainerDied","Data":"6057cbbbdcf5dd8a5b4663e9b19438908528c754dea02cccba1e6a55ad8e092f"} Jul 07 18:08:01.544482 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.544464 2579 scope.go:117] "RemoveContainer" containerID="728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f" Jul 07 18:08:01.553169 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.553150 2579 scope.go:117] "RemoveContainer" containerID="728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f" Jul 07 18:08:01.553501 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:08:01.553477 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f\": container with ID starting with 728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f not found: ID does not exist" containerID="728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f" Jul 07 18:08:01.553593 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.553506 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f"} err="failed to get container status \"728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f\": rpc error: code = NotFound desc = could not find container \"728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f\": container with ID starting with 728d81dae2bf9b9a004dc2f14ad93873d029496e4a7fc351d4952af1f37e9e4f not found: ID does not exist" Jul 07 18:08:01.574201 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.574181 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-dshm\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:01.574317 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.574205 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-home\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:01.574317 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.574217 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tls-certs\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:01.574317 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.574229 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-model-cache\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:01.574317 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.574241 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-tmp-dir\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:01.574317 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.574252 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ckpg\" (UniqueName: \"kubernetes.io/projected/7cce29dd-a243-4d8b-ae94-1af8d216ebaf-kube-api-access-8ckpg\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:01.578611 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.578581 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f"] Jul 07 18:08:01.579917 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.579897 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1e464b/rate-test-model-1e464b-kserve-59cdf77899-dq52f"] Jul 07 18:08:01.842905 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:01.842872 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cce29dd-a243-4d8b-ae94-1af8d216ebaf" path="/var/lib/kubelet/pods/7cce29dd-a243-4d8b-ae94-1af8d216ebaf/volumes" Jul 07 18:08:33.022146 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.022113 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf"] Jul 07 18:08:33.022673 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.022328 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" podUID="6797a991-1dfa-4cff-bef6-3d9be7371b3b" containerName="maas-api" containerID="cri-o://508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145" gracePeriod=30 Jul 07 18:08:33.274406 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.274341 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:08:33.337381 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.337352 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6797a991-1dfa-4cff-bef6-3d9be7371b3b-openshift-service-ca\") pod \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " Jul 07 18:08:33.337526 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.337404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6797a991-1dfa-4cff-bef6-3d9be7371b3b-maas-api-tls\") pod \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " Jul 07 18:08:33.337526 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.337439 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jgrp\" (UniqueName: \"kubernetes.io/projected/6797a991-1dfa-4cff-bef6-3d9be7371b3b-kube-api-access-9jgrp\") pod \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\" (UID: \"6797a991-1dfa-4cff-bef6-3d9be7371b3b\") " Jul 07 18:08:33.337724 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.337700 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6797a991-1dfa-4cff-bef6-3d9be7371b3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6797a991-1dfa-4cff-bef6-3d9be7371b3b" (UID: "6797a991-1dfa-4cff-bef6-3d9be7371b3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:08:33.339598 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.339570 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6797a991-1dfa-4cff-bef6-3d9be7371b3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6797a991-1dfa-4cff-bef6-3d9be7371b3b" (UID: "6797a991-1dfa-4cff-bef6-3d9be7371b3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:08:33.339715 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.339604 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6797a991-1dfa-4cff-bef6-3d9be7371b3b-kube-api-access-9jgrp" (OuterVolumeSpecName: "kube-api-access-9jgrp") pod "6797a991-1dfa-4cff-bef6-3d9be7371b3b" (UID: "6797a991-1dfa-4cff-bef6-3d9be7371b3b"). InnerVolumeSpecName "kube-api-access-9jgrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:08:33.438963 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.438937 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6797a991-1dfa-4cff-bef6-3d9be7371b3b-openshift-service-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:33.438963 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.438961 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6797a991-1dfa-4cff-bef6-3d9be7371b3b-maas-api-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:33.439115 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.438971 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jgrp\" (UniqueName: \"kubernetes.io/projected/6797a991-1dfa-4cff-bef6-3d9be7371b3b-kube-api-access-9jgrp\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\"" Jul 07 18:08:33.689295 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.689193 2579 generic.go:356] "Generic (PLEG): container finished" podID="6797a991-1dfa-4cff-bef6-3d9be7371b3b" containerID="508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145" exitCode=0 Jul 07 18:08:33.689295 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.689241 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" Jul 07 18:08:33.689295 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.689254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" event={"ID":"6797a991-1dfa-4cff-bef6-3d9be7371b3b","Type":"ContainerDied","Data":"508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145"} Jul 07 18:08:33.689608 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.689328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf" event={"ID":"6797a991-1dfa-4cff-bef6-3d9be7371b3b","Type":"ContainerDied","Data":"9beb9b6dc01fe2772f078503801e54942fbc805e83b05ca8f1862694fdcf1ced"} Jul 07 18:08:33.689608 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.689354 2579 scope.go:117] "RemoveContainer" containerID="508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145" Jul 07 18:08:33.697823 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.697805 2579 scope.go:117] "RemoveContainer" containerID="508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145" Jul 07 18:08:33.698069 ip-10-0-129-95 kubenswrapper[2579]: E0707 18:08:33.698051 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145\": container with ID starting with 508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145 not found: ID does not exist" containerID="508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145" Jul 07 18:08:33.698131 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.698082 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145"} err="failed to get container status \"508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145\": rpc error: code = NotFound desc = could not find container \"508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145\": container with ID starting with 508349990b97f3f8cce35e2c6470ed23d526459ac06c3c98bc8aefaa02c0c145 not found: ID does not exist" Jul 07 18:08:33.726571 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.726539 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf"] Jul 07 18:08:33.737631 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.737608 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-df746e-7cff6f499c-gm2gf"] Jul 07 18:08:33.841647 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:33.841620 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6797a991-1dfa-4cff-bef6-3d9be7371b3b" path="/var/lib/kubelet/pods/6797a991-1dfa-4cff-bef6-3d9be7371b3b/volumes" Jul 07 18:08:47.208630 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:47.208602 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-2ddkd_3b6e072c-8116-4676-a8fd-caf8e707f408/copy-manifests/0.log" Jul 07 18:08:47.218885 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:47.218847 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-2ddkd_3b6e072c-8116-4676-a8fd-caf8e707f408/manager/0.log" Jul 07 18:08:50.503709 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:50.503681 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6zh2s_f6332754-e093-42fe-95dd-36741751618a/discovery/0.log" Jul 07 18:08:50.631481 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:50.631456 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668c694757-ftrl5_29d9be73-4207-4fa0-99a9-5dba5872ae20/kube-auth-proxy/0.log" Jul 07 18:08:52.011473 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:52.011434 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwds5v_fd684d89-b605-4a33-b94a-2f1002662d2a/main/0.log" Jul 07 18:08:59.756491 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:08:59.756457 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-29cf4_de81cac2-c315-4825-927b-2b4c4f8f3790/global-pull-secret-syncer/0.log" Jul 07 18:09:00.032226 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:00.032151 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ttk6q_509edf3c-5fd7-4e16-85b4-7405c98a2900/konnectivity-agent/0.log" Jul 07 18:09:00.059943 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:00.059914 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-95.ec2.internal_eff5104d7452b5ed4a3bf2742a0cf695/haproxy/0.log" Jul 07 18:09:07.986865 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:07.986835 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rjwjb_4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f/kube-state-metrics/0.log" Jul 07 18:09:08.014225 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.014201 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rjwjb_4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f/kube-rbac-proxy-main/0.log" Jul 07 18:09:08.054034 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.054006 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rjwjb_4e46accb-4fa2-4626-bf00-0c8b9b4c5b6f/kube-rbac-proxy-self/0.log" Jul 07 18:09:08.133576 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.133541 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-cxgql_c023f6ac-d192-4c18-8d9e-04d029798295/monitoring-plugin/0.log" Jul 07 18:09:08.449502 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.449421 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnjpm_7ba304d4-350f-4117-847c-c3ec3fd08f78/node-exporter/0.log" Jul 07 18:09:08.486957 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.486932 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnjpm_7ba304d4-350f-4117-847c-c3ec3fd08f78/kube-rbac-proxy/0.log" Jul 07 18:09:08.528787 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.528769 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnjpm_7ba304d4-350f-4117-847c-c3ec3fd08f78/init-textfile/0.log" Jul 07 18:09:08.583475 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.583453 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9nlxs_de194883-e38b-41b6-b9ec-aec7ffe6342b/kube-rbac-proxy-main/0.log" Jul 07 18:09:08.636697 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.636666 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9nlxs_de194883-e38b-41b6-b9ec-aec7ffe6342b/kube-rbac-proxy-self/0.log" Jul 07 18:09:08.681060 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.681039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9nlxs_de194883-e38b-41b6-b9ec-aec7ffe6342b/openshift-state-metrics/0.log" Jul 07 18:09:08.739930 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.739905 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7fd3faf-15bd-4af7-bb5b-617b476c782a/prometheus/0.log" Jul 07 18:09:08.783039 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.783019 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7fd3faf-15bd-4af7-bb5b-617b476c782a/config-reloader/0.log" Jul 07 18:09:08.817094 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.817076 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7fd3faf-15bd-4af7-bb5b-617b476c782a/thanos-sidecar/0.log" Jul 07 18:09:08.861016 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.860996 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7fd3faf-15bd-4af7-bb5b-617b476c782a/kube-rbac-proxy-web/0.log" Jul 07 18:09:08.897037 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.897012 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7fd3faf-15bd-4af7-bb5b-617b476c782a/kube-rbac-proxy/0.log" Jul 07 18:09:08.937553 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.937529 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7fd3faf-15bd-4af7-bb5b-617b476c782a/kube-rbac-proxy-thanos/0.log" Jul 07 18:09:08.994579 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:08.994507 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7fd3faf-15bd-4af7-bb5b-617b476c782a/init-config-reloader/0.log" Jul 07 18:09:09.203976 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.203954 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57fccdd65f-r8qmc_4514d7be-35de-4eac-b3c1-0b9b3a41f9bf/telemeter-client/0.log" Jul 07 18:09:09.239159 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.239138 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57fccdd65f-r8qmc_4514d7be-35de-4eac-b3c1-0b9b3a41f9bf/reload/0.log" Jul 07 18:09:09.282989 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.282923 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57fccdd65f-r8qmc_4514d7be-35de-4eac-b3c1-0b9b3a41f9bf/kube-rbac-proxy/0.log" Jul 07 18:09:09.350365 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.350336 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/thanos-query/0.log" Jul 07 18:09:09.412191 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.412159 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy-web/0.log" Jul 07 18:09:09.479161 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.479135 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy/0.log" Jul 07 18:09:09.525452 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.525423 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/prom-label-proxy/0.log" Jul 07 18:09:09.564582 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.564501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy-rules/0.log" Jul 07 18:09:09.592767 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.592738 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-789dd4f6c8-7lphr_f4b1a2ef-8141-449c-99bf-0a2bd48b1ade/kube-rbac-proxy-metrics/0.log" Jul 07 18:09:09.803850 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.803821 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5"] Jul 07 18:09:09.804146 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804135 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cce29dd-a243-4d8b-ae94-1af8d216ebaf" containerName="main" Jul 07 18:09:09.804192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804148 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cce29dd-a243-4d8b-ae94-1af8d216ebaf" containerName="main" Jul 07 18:09:09.804192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804159 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" containerName="maas-api" Jul 07 18:09:09.804192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804164 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" containerName="maas-api" Jul 07 18:09:09.804192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804173 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405ff7bd-9213-4e20-a925-3400ca8d138a" containerName="main" Jul 07 18:09:09.804192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804177 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="405ff7bd-9213-4e20-a925-3400ca8d138a" containerName="main" Jul 07 18:09:09.804192 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804192 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6797a991-1dfa-4cff-bef6-3d9be7371b3b" containerName="maas-api" Jul 07 18:09:09.804404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804199 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6797a991-1dfa-4cff-bef6-3d9be7371b3b" containerName="maas-api" Jul 07 18:09:09.804404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804254 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="856b46a6-b800-4a2e-aed7-d5ca0a62c2d9" containerName="maas-api" Jul 07 18:09:09.804404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804284 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="405ff7bd-9213-4e20-a925-3400ca8d138a" containerName="main" Jul 07 18:09:09.804404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804291 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cce29dd-a243-4d8b-ae94-1af8d216ebaf" containerName="main" Jul 07 18:09:09.804404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.804299 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6797a991-1dfa-4cff-bef6-3d9be7371b3b" containerName="maas-api" Jul 07 18:09:09.807330 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.807315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:09.809821 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.809801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qlcl8\"/\"kube-root-ca.crt\"" Jul 07 18:09:09.810678 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.810654 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qlcl8\"/\"default-dockercfg-xx92f\"" Jul 07 18:09:09.810799 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.810734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qlcl8\"/\"openshift-service-ca.crt\"" Jul 07 18:09:09.818035 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.817991 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5"] Jul 07 18:09:09.936186 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.936148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-sys\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:09.936433 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.936193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-podres\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:09.936433 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.936222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-proc\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:09.936433 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.936339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-kube-api-access-zjxlf\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:09.936433 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:09.936380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-lib-modules\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037254 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-podres\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037254 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-proc\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-kube-api-access-zjxlf\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-lib-modules\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037424 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-proc\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-sys\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-podres\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037483 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-sys\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.037718 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.037520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-lib-modules\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.049305 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.049259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/0cf07b64-8b13-4cf9-a33f-8b632d1b5287-kube-api-access-zjxlf\") pod \"perf-node-gather-daemonset-fqpn5\" (UID: \"0cf07b64-8b13-4cf9-a33f-8b632d1b5287\") " pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.117602 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.117518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.253853 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.253776 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5"] Jul 07 18:09:10.256456 ip-10-0-129-95 kubenswrapper[2579]: W0707 18:09:10.256429 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0cf07b64_8b13_4cf9_a33f_8b632d1b5287.slice/crio-5cfc84e16bd510135259e70efb13e45dd83208a6ae92551a83e75ffc8df9fbc6 WatchSource:0}: Error finding container 5cfc84e16bd510135259e70efb13e45dd83208a6ae92551a83e75ffc8df9fbc6: Status 404 returned error can't find the container with id 5cfc84e16bd510135259e70efb13e45dd83208a6ae92551a83e75ffc8df9fbc6 Jul 07 18:09:10.853138 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.853109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" event={"ID":"0cf07b64-8b13-4cf9-a33f-8b632d1b5287","Type":"ContainerStarted","Data":"bac5e3d686465ba1f69dfbc3108d79f563ba12d6ef289d2884b6a31ec30208d7"} Jul 07 18:09:10.853404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.853155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" event={"ID":"0cf07b64-8b13-4cf9-a33f-8b632d1b5287","Type":"ContainerStarted","Data":"5cfc84e16bd510135259e70efb13e45dd83208a6ae92551a83e75ffc8df9fbc6"} Jul 07 18:09:10.853404 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.853188 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:10.879195 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:10.879147 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" podStartSLOduration=1.879131559 podStartE2EDuration="1.879131559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:09:10.876863949 +0000 UTC m=+2785.717177621" watchObservedRunningTime="2026-07-07 18:09:10.879131559 +0000 UTC m=+2785.719445260" Jul 07 18:09:14.228604 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:14.228567 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vsgtq_e72ea041-019a-426c-8dd3-404f28a1f764/dns/0.log" Jul 07 18:09:14.264515 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:14.264480 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vsgtq_e72ea041-019a-426c-8dd3-404f28a1f764/kube-rbac-proxy/0.log" Jul 07 18:09:14.322997 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:14.322972 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mw4xf_955bef82-3b6a-47dd-9a19-113ab9aaebee/dns-node-resolver/0.log" Jul 07 18:09:15.046014 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:15.045984 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fhpv8_368511c4-3427-4707-9c1f-7e5d31d53fb4/node-ca/0.log" Jul 07 18:09:16.368873 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:16.368846 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6zh2s_f6332754-e093-42fe-95dd-36741751618a/discovery/0.log" Jul 07 18:09:16.400063 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:16.400038 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668c694757-ftrl5_29d9be73-4207-4fa0-99a9-5dba5872ae20/kube-auth-proxy/0.log" Jul 07 18:09:16.867801 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:16.867777 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qlcl8/perf-node-gather-daemonset-fqpn5" Jul 07 18:09:17.606492 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:17.606461 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z25lv_f0b5daea-5f8c-4e4e-8bfd-e4f21ea08518/serve-healthcheck-canary/0.log" Jul 07 18:09:18.510004 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:18.509960 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jtrzj_18b4b43e-4449-4593-8af1-eb189b662584/kube-rbac-proxy/0.log" Jul 07 18:09:18.545794 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:18.545768 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jtrzj_18b4b43e-4449-4593-8af1-eb189b662584/exporter/0.log" Jul 07 18:09:18.594208 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:18.594172 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jtrzj_18b4b43e-4449-4593-8af1-eb189b662584/extractor/0.log" Jul 07 18:09:20.944286 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:20.944233 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6zh2s_f6332754-e093-42fe-95dd-36741751618a/discovery/0.log" Jul 07 18:09:20.986458 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:20.986430 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668c694757-ftrl5_29d9be73-4207-4fa0-99a9-5dba5872ae20/kube-auth-proxy/0.log" Jul 07 18:09:21.751483 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:21.751449 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-2ddkd_3b6e072c-8116-4676-a8fd-caf8e707f408/manager/0.log" Jul 07 18:09:21.795885 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:21.795861 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684cdbc7fc-2ddkd_3b6e072c-8116-4676-a8fd-caf8e707f408/copy-manifests/0.log" Jul 07 18:09:22.598076 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:22.598047 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6zh2s_f6332754-e093-42fe-95dd-36741751618a/discovery/0.log" Jul 07 18:09:22.612421 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:22.612386 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-668c694757-ftrl5_29d9be73-4207-4fa0-99a9-5dba5872ae20/kube-auth-proxy/0.log" Jul 07 18:09:23.584042 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:23.584014 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-686dcd8c67-qvhsw_8eb9601b-1989-4605-b25e-c794bf0723b2/manager/0.log" Jul 07 18:09:23.640952 ip-10-0-129-95 kubenswrapper[2579]: I0707 18:09:23.640929 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gzpd4_24b2c9d3-aaa3-4037-bf4c-2f62f08e5048/openshift-lws-operator/0.log"